OpenAI GPT-Live

Use openai:live:gpt-live-1 to evaluate OpenAI's GPT-Live API. It connects to /v1/live/sessions and supports full-duplex audio, where the model can listen and speak simultaneously. openai:gpt-live-1 and openai:live select the same provider, as do dated openai:gpt-live-1-YYYY-MM-DD snapshots. The official OpenAI endpoint requires a gpt-live-* model; compatible gateways can use custom model names. The transcription-only gpt-live-transcribe model requires a separate transcription session and is not supported here.

Set OPENAI_API_KEY to an OpenAI project key with Live access. For a compatible gateway, set apiBaseUrl and authenticate with apiKey, a credential header in headers (such as Authorization or api-key), or userinfo in apiBaseUrl, which is sent as a Basic Authorization header. When a gateway uses its own credential header or URL userinfo, an OPENAI_API_KEY from the environment is not sent to it unless the provider or prompt config sets apiKey or apiKeyEnvar. A prompt's config can set its own apiBaseUrl, apiHost, organization, and credentials; each session uses the merged endpoint, default headers, and credentials together. For gateways that authenticate with an opaque header name such as X-Session-Access, set apiKeyRequired: false; the header value is still redacted from diagnostics.

Quickstart

A prompt that overrides the endpoint must supply its required credential headers in that prompt's config.headers. Credential headers from the provider configuration are not inherited by endpoint overrides.

promptfooconfig.yaml

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: GPT-Live spoken answers
prompts:
  - 'What is the capital of France? Answer in one sentence.'
providers:
  - id: openai:live:gpt-live-1
    config:
      instructions: Keep answers short. Delegate questions needing research to the backend.
      audio:
        output:
          voice: marin
      responseWindowMs: 30000
      delegation:
        type: responses
        responses:
          model: gpt-5.6-luna
          instructions: Return concise, factual answers.
tests:
  - assert:
      - type: contains
        value: Paris

Run npx promptfoo@latest eval --no-cache -o results.json. Assertions evaluate the assistant's transcript. The result also includes playable audio and timestamped user and assistant transcript fragments. Fragments keep their original spacing, so a transcript can begin with a space; prefer contains or icontains to equals.

For native audio grading with llm-rubric, use an audio-capable Chat grader such as openai:chat:gpt-audio-1.5. GPT-Live used as a grader receives the transcript; it does not accept the MP3 and arbitrary WAV files supported by Chat audio grading.

Each test creates a new session. Text prompts seed a user message in startup history, then request a spoken answer while streaming silence. This is useful for checking answer content; use recorded audio to evaluate speech recognition, pauses, or interruptions.

Audio input

Use OpenAI chat-format audio content in your prompt:

[
  {
    "role": "user",
    "content": [
      {
        "type": "input_audio",
        "input_audio": { "data": "{{audio}}", "format": "wav" }
      }
    ]
  }
]

Set the audio test variable to file://sample.wav. WAV files must contain mono, signed 16-bit PCM matching audio.format.rate (24,000 Hz by default). The file must declare one valid format before its audio data, and each data chunk must contain complete PCM16 samples. Promptfoo removes the WAV container before streaming and ignores bytes after its declared RIFF boundary. It accepts streaming headers with unknown sizes, such as OpenAI text-to-speech wav output and ffmpeg pipe output, and WAVE_FORMAT_EXTENSIBLE PCM. It rejects mismatched rates and compressed formats; it does not resample audio. Base64 input may include or omit padding.

Raw base64 audio accepts pcm16, g711_ulaw, or g711_alaw in input_audio.format. Configure the corresponding shared input/output format:

audio.format	Input encoding
{type: audio/pcm, rate: 24000}	pcm16 or matching wav
{type: audio/pcm, rate: 16000}	pcm16 or matching wav
{type: audio/pcmu, rate: 8000}	g711_ulaw
{type: audio/pcma, rate: 8000}	g711_alaw

Audio is accepted only in the final user message. Supply prior history as text messages. System messages map to Live's developer role; user text stays user content. Images are not accepted by the Live voice frontend.

Capture duration

Promptfoo streams audio in 20 ms frames at the configured sample rate. Recorded audio is followed by responseWindowMs of silence (default: 30 seconds). For text prompts, Promptfoo streams silence immediately, asks Live to answer, and starts the response window when Live acknowledges that instruction. It then sends session.close and waits for final usage. The input clip plus response window may total at most five minutes.

Live has no authoritative speech-completed event. The response window is a fixed recording window and may cut off speech. Increase it for long replies or backend work. Backend completion and transcript gaps do not end the capture early. Transcript and audio deltas received after closing begins are ignored; final usage and errors are still processed. Output audio contains the received samples; transcript timestamps are on the session timeline and do not establish playback timing.

Output audio is capped by the configured format and capture duration, with a maximum of five minutes. Startup delay does not increase this budget. Excess audio ends the capture with an error.

websocketTimeout covers the handshake, session.started, and a text prompt's acknowledgment (default: 30 seconds); closeTimeoutMs controls finalization (default: 15 seconds). These timeouts plus the capture duration must fit within REQUEST_TIMEOUT_MS (default: five minutes). Increase it for a full five-minute capture. Cancellation and eval shutdown release active sockets. Live responses are not cached.

Backend delegation

For managed Responses delegation, set delegation.type: responses and delegation.responses.model to a nonempty string of at most 256 UTF-8 bytes. The backend can use function and web_search tools. It has its own instructions, token limit, reasoning settings, and service tier. Follow OpenAI's delegation configuration for supported settings.

For custom functions, set functionCallHandler: file://tools.js. Export an async function (name, args, signal) => string, where args is the JSON argument string. Promptfoo checks the function name against configured tools, collects completed calls, returns every result, and then continues the backend response. Handlers must enforce permissions for actions they execute.

maxToolIterations separately limits each session's function calls and distinct delegations, including client and managed Responses delegations (default: 8 each, allowed range 1 to 64). Repeated delegation IDs do not count again. When either limit is exceeded, Promptfoo ends the capture and reports an error without invoking another client handler. A managed Responses delegation may already be running when its creation event reaches Promptfoo.

Function-call IDs and names must be nonempty and are limited to 256 bytes each. Their combined size, including argument strings, is limited to 1 MiB across the session; exceeding this limit ends the capture before the call is buffered or executed.

Each function-handler result is limited to 1 MiB before it is serialized or sent. Return a summary or a reference for larger results.

For your own model or agent harness, use delegation.type: client and delegationHandler: file://backend.js:

export default async function handleDelegation(request, signal) {
  // request: { id, offsetMs, input, transcript }
  // Consult your backend using the startup history and timestamped transcripts.
  // Honor signal to stop work when the eval ends.
  return 'The order shipped today.';
}

The delegation event contains an ID, not task text. Your handler receives a snapshot of the conversation collected so far. Return a concise string within Live's 500-token append limit. Promptfoo sends it as commentary using the original delegation ID. Handler errors are reported without sending exception details to the model. Late results are discarded after closing.

Promptfoo rejects client-handler results larger than 64 KiB before sending them. This bounds local buffering; the gateway still enforces the 500-token limit.

Conversation copies retained by pending client handlers are limited to 8 MiB of text and 50,000 input/transcript entries in total. Complete handlers promptly or reduce conversation history if this limit is reached. The budget is released when each handler settles, including after cancellation.

Omitting delegation selects client mode. If Live requests backend work without a handler, the eval reports an error. Backend work that is pending when the capture ends, or requested after it ends, also reports an error.

A spoken answer does not show that backend work ran. metadata.delegations lists each delegation's id, target, and offsetMs, and metadata.backendResponses lists completed Responses calls. Known credentials echoed in session IDs, backend metadata, or close reasons are redacted. Assert on these fields when a test requires delegation:

assert:
  - type: javascript
    value: context.providerResponse.metadata.delegations.length > 0

Errors and refusals

Startup failures report the server's error code and message. After startup, Live error events are recorded in metadata.apiErrors with the code, type, redacted message, parameter, and rejected client event, without ending the capture:

• Moderation and safety errors are graded as refusals, including when they reject one of promptfoo's commands or arrive while the session is closing.
• A rejected opening instruction or commentary ends the capture, because Live was never asked to speak.
• Other rejected commands and unattributed errors mark the result as an error and keep the full transcript.
• Errors for pending commands that session.close cancels are recorded but do not change the result.

Safety interventions are graded as refusals rather than errors. A moderation error sets isRefusal and guardrails.flagged. A session closed by the safety filter (reason: content) also sets finishReason: content_filter and conversationEnded, so multi-turn red team strategies stop probing. Sessions that close as expired or connection_lost remain errors.

Results and cost

output is the assistant transcript, concatenated exactly as received. Transcripts preserve model content for grading, including sensitive content returned by the endpoint. audio contains playable PCM16 WAV at the session's sample rate, including decoded G.711 responses. metadata.transcript retains both speakers' fragments and start_ms/end_ms timestamps, including overlap. The request count includes the Live session and each completed backend response.

metadata.voiceSeconds is the latest valid cumulative usage snapshot. Usage must be finite, nonnegative, and no greater than REQUEST_TIMEOUT_MS rounded up to seconds. metadata.finalUsageConfirmed is true only when session.closed supplies valid final usage. A dropped connection preserves partial output and observed usage, and reports an error. Numeric overflow in voice, backend, or total cost reports an error and omits the affected cost.

Voice cost uses the published $0.05/minute rate for gpt-live-1 and its dated snapshots, such as gpt-live-1-2026-09-01. Set costPerMinute to override it or to price other models. Responses token usage and model cost are accumulated separately from nested backend events. cost includes confirmed voice and backend model costs, excluding hosted-tool fees. Client-managed backend costs are unknown, so those sessions expose voice cost in metadata.voiceCost and omit total cost.

See the runnable example and OpenAI's Live session guide.