Providers | promptfoo

📄️ OpenAI

To use the OpenAI API, set the OPENAIAPIKEY environment variable, specify via apiKey field in the configuration file or pass the API key as an argument to the constructor.

📄️ Anthropic

This provider supports the Anthropic Claude series of models.

📄️ Azure

The azureopenai provider is an interface to OpenAI through Azure. It behaves the same as the OpenAI provider.

The Echo Provider is a simple utility provider that returns the input prompt as the output. It's particularly useful for testing, debugging, and validating pre-generated outputs without making any external API calls.

📄️ Llama.cpp

The llama provider is compatible with the HTTP server bundled with llama.cpp. This allows you to leverage the power of llama.cpp models within Promptfoo.

📄️ Ollama

The ollama provider is compatible with Ollama, which enables access to Llama, Mixtral, Mistral, and more.

📄️ Google Vertex

The vertex provider is compatible with Google's Vertex AI offering, which offers access to models such as Gemini and Bison.

📄️ IBM BAM

The bam provider integrates with IBM's BAM API, allowing access to various models like meta-llama/llama-2-70b-chat and ibm/granite-13b-chat-v2.

📄️ Google AI Studio

The google provider is compatible with Google AI Studio (formerly known as PaLM), which offers access to Gemini models.

📄️ Generic webhook

The webhook provider can be useful for triggering more complex flows or prompt chains end to end in your app.

📄️ Custom Javascript

To create a custom API provider, implement the ApiProvider interface in a separate module. Here is the interface:

📄️ Custom Python

The python provider allows you to use a Python script as an API provider for evaluating prompts. This is useful when you have custom logic or models implemented in Python that you want to integrate with your test suite.

📄️ Custom scripts

You may use any shell command as an API provider. This is particularly useful when you want to use a language or framework that is not directly supported by promptfoo.

📄️ HTTP API

Setting provider id to a URL sends an HTTP request to the endpoint. This is a general-purpose way to use any HTTP endpoint for inference.

📄️ HuggingFace

promptfoo includes support for the HuggingFace Inference API, for text generation, classification, and embeddings related tasks.

📄️ LocalAI (Llama, Alpaca, GPT4All, ...)

LocalAI is an API wrapper for open-source LLMs that is compatible with OpenAI. You can run LocalAI for compatibility with Llama, Alpaca, Vicuna, GPT4All, RedPajama, and many other models compatible with the ggml format.

📄️ Replicate

Replicate is an API for machine learning models. It currently hosts models like Llama v2, Gemma, and Mistral/Mixtral.

📄️ AI21 Labs

The AI21 Labs API offers access to AI21 models such as jamba-1.5-mini and jamba-1.5-large.

📄️ Bedrock

The bedrock lets you use Amazon Bedrock in your evals. This is a common way to access Anthropic's Claude and other models. The complete list of available models can be found here.

📄️ Cloudflare Workers AI

This provider supports the models provided by Cloudflare Workers AI, a serverless edge embedding and inference runtime.

📄️ Cohere

The cohere provider is an interface to Cohere AI's chat inference API, with models such as Command R that are optimized for RAG and tool usage.

📄️ Groq

The Groq API is integrated into promptfoo using the Groq SDK, providing a native experience for using Groq models in your evaluations. Groq offers high-performance inference for various large language models.

📄️ LiteLLM

The LiteLLM provides access to hundreds of LLMs.

📄️ Manual Input

The Manual Input Provider allows you to manually enter responses for each prompt during the evaluation process. This can be useful for testing, debugging, or when you want to provide custom responses without relying on an automated API.