AI21 Labs

The AI21 Labs API offers access to AI21 models such as jamba-1.5-mini and jamba-1.5-large.

API Key

To use AI21 Labs, you need to set the AI21_API_KEY environment variable, or specify the apiKey in the provider configuration.

Example of setting the environment variable:

export AI21_API_KEY=your_api_key_here

Model Selection

You can specify which AI21 model to use in your configuration. Currently, the following models are available:

1. jamba-1.5-mini
2. jamba-1.5-large

Here's an example config that compares AI21 models:

providers:
  - ai21:jamba-1.5-mini
  - ai21:jamba-1.5-large

Options

The AI21 provider supports several options to customize the behavior of the model. These include:

• temperature: Controls the randomness of the output.
• top_p: Controls nucleus sampling, affecting the randomness of the output.
• max_tokens: The maximum length of the generated text.
• response_format: Set to { type: 'json_object' } for JSON output or { type: 'text' } for text output.
• apiKeyEnvar: An environment variable that contains the API key.
• apiBaseUrl: The base URL of the AI21 API.

Example Configuration

Here's an example configuration for the AI21 provider:

providers:
  - ai21:jamba-1.5-mini
    config:
      apiKey: your_api_key_here
      temperature: 0.1
      top_p: 1
      max_tokens: 1024
      response_format: { type: 'json_object' }

This configuration uses the jamba-1.5-mini model with a temperature of 0.1, top-p sampling with a value of 1, a maximum output length of 1024 tokens, and JSON-formatted output.

Cost

The cost of using AI21 models depends on the model and the number of input and output tokens. Here are the costs for the available models:

• jamba-1.5-mini: $0.2 per 1M input tokens, $0.4 per 1M output tokens
• jamba-1.5-large: $2 per 1M input tokens, $8 per 1M output tokens

You can override promptfoo's built-in pricing with inputCost and outputCost in the provider configuration. The legacy cost option still works as a shared fallback when you want the same rate for both directions.

Supported environment variables

These AI21-related environment variables are supported:

Variable	Description
AI21_API_BASE_URL	The base URL (protocol + hostname + port) to use for the AI21 API.
AI21_API_KEY	AI21 API key.