OpenAI ChatKit

ChatKit provides a chat interface for an agent application. Use Promptfoo to evaluate the agent or workflow behind that interface. OpenAI's custom ChatKit integration guide describes how to connect ChatKit to a backend you run.

Choose a provider that matches how your application runs:

Application	Promptfoo integration
JavaScript or TypeScript SDK agent	OpenAI Agents SDK provider (openai:agents:*)
Python Agents SDK workflow	Agents SDK Python guide
Workflow with custom orchestration	JavaScript/TypeScript provider or Python provider
Application exposed through an HTTP endpoint	HTTP provider

For web UI tests, configure the Browser provider with application-specific interaction and extraction steps. Specialized browser automation can run through a custom JavaScript provider using Playwright.

Migration

The legacy openai:chatkit provider is unsupported. Configurations that use it fail with a migration error before starting a browser or calling OpenAI. Replace each openai:chatkit entry with an integration for your application.

For Agent Builder workflows, follow OpenAI's migration guide to export your workflow as Agents SDK code.

An exported workflow may contain several agents, branching logic, approval steps, and state management. Use a custom provider or HTTP endpoint to invoke the complete workflow when loading a single SDK agent would omit that behavior. A workflow ID alone cannot be substituted into openai:agents:*.

Carry over your prompts and assertions, and map conversation state, tool approvals, and timeouts to the replacement integration. Legacy provider options such as workflowId, version, approvalHandling, and usePool do not transfer automatically.

Run representative single-turn and multi-turn tests against the replacement before relying on its results.