Skip to main content

TrueFoundry

TrueFoundry routes requests to LLM providers and MCP servers. Promptfoo uses its OpenAI-compatible API for chat completions and embeddings.

The TrueFoundry provider supports:

  • Chat completions from multiple LLM providers (OpenAI, Anthropic, Google Gemini, Groq, Mistral, and more)
  • Embeddings
  • Tool use and function calling
  • MCP (Model Context Protocol) servers
  • Custom metadata and logging configuration
  • Real-time observability and monitoring

Setup​

To use TrueFoundry, you need to set up your API key:

  1. Create a TrueFoundry account and obtain an API key from the TrueFoundry Console.
  2. Set the TRUEFOUNDRY_API_KEY environment variable:
export TRUEFOUNDRY_API_KEY=your_api_key_here

Alternatively, you can specify the apiKey in the provider configuration (see below).

Configuration​

Configure the TrueFoundry provider in your promptfoo configuration file. The model name should follow the format provider-account/model-name (e.g., openai-main/gpt-5):

promptfooconfig.yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: truefoundry:openai-main/gpt-5
    config:
      temperature: 0.7
      max_completion_tokens: 100
prompts:
  - Write a funny tweet about {{topic}}
tests:
  - vars:
      topic: cats
  - vars:
      topic: dogs

Note: The model identifier format is provider-account/model-name. The provider-account is the name of your provider integration in TrueFoundry (e.g., openai-main, anthropic-main). You can find available models in the TrueFoundry LLM Playground UI.

Basic Configuration Options​

The TrueFoundry provider supports the following configuration options:

  • task: Set chat or embedding explicitly. When omitted, model IDs containing embedding select embeddings; other IDs select chat. The full account/model ID is sent unchanged.
  • temperature: Controls randomness in output between 0 and 2
  • max_tokens: Maximum number of tokens to generate
  • max_completion_tokens: Maximum number of tokens that can be generated in the chat completion
  • top_p: Alternative to temperature sampling using nucleus sampling
  • presence_penalty: Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far
  • frequency_penalty: Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far
  • stop: Up to 4 sequences where the API will stop generating further tokens
  • response_format: Object specifying the format that the model must output (e.g., JSON mode)
  • seed: For deterministic sampling (best effort)

Custom API Base URL​

For self-hosted or enterprise deployments, you can specify a custom API base URL:

providers:
  - id: truefoundry:openai-main/gpt-5
    config:
      apiBaseUrl: 'https://your-custom-gateway.example.com'
      temperature: 0.7

If not specified, the default URL https://llm-gateway.truefoundry.com is used.

TrueFoundry-Specific Configuration​

TrueFoundry provides additional configuration options for metadata tracking, logging, and local OpenAI cost lookup:

providers:
  - id: truefoundry:production-east/gpt-5
    config:
      temperature: 0.7
      openaiAccountNames:
        - production-east
      metadata:
        user_id: 'test-user'
        environment: 'production'
        custom_key: 'custom_value'
      loggingConfig:
        enabled: true

Configuration options:

  • metadata: Custom metadata to track with each request (object with key-value pairs)
  • loggingConfig: Logging configuration for observability (must include enabled: true)
  • openaiAccountNames: Additional TrueFoundry OpenAI account names to use for local OpenAI cost lookup. openai-main is recognized by default. This option does not rewrite the model ID sent to TrueFoundry.

Model Support​

TrueFoundry supports models from multiple providers. Use the format provider-account/model-name:

OpenAI Models​

providers:
  - truefoundry:openai-main/gpt-5
  - truefoundry:openai-main/gpt-4o
  - truefoundry:openai-main/gpt-4o-mini
  - truefoundry:openai-main/o1

Anthropic Models​

For Claude 5, set omitDefaults: true to omit Promptfoo's default temperature: 0. Leave sampling parameters unset in your config and environment; explicit values still apply.

providers:
  - id: truefoundry:anthropic-main/claude-sonnet-5
    config:
      omitDefaults: true
  - id: truefoundry:anthropic-main/claude-opus-5
    config:
      omitDefaults: true
  - truefoundry:anthropic-main/claude-haiku-4-5

Google Gemini Models​

providers:
  - truefoundry:vertex-ai-main/gemini-2.5-pro
  - truefoundry:vertex-ai-main/gemini-2.5-flash
  - truefoundry:vertex-ai-main/gemini-1.5-pro

Other Providers​

providers:
  - truefoundry:groq-main/llama-3.3-70b-versatile
  - truefoundry:mistral-main/mistral-large-latest

Embeddings​

Use task: embedding to select the embeddings API for any account/model ID, including custom aliases. Configure it as the embedding provider for a similar assertion:

promptfooconfig.yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
prompts:
  - '{{query}}'
providers:
  - echo
defaultTest:
  options:
    provider:
      embedding:
        id: truefoundry:openai-main/text-embedding-3-large
        config:
          task: embedding
          metadata:
            user_id: 'embedding-test'
          loggingConfig:
            enabled: true
tests:
  - vars:
      query: 'What is machine learning?'
    assert:
      - type: similar
        value: 'How does machine learning work?'
        threshold: 0.8

Cohere Embeddings​

When using Cohere models, select the embedding task and send the required input_type through passthrough. For example, replace the embedding provider above with:

defaultTest:
  options:
    provider:
      embedding:
        id: truefoundry:cohere-main/embed-english-v3.0
        config:
          task: embedding
          passthrough:
            input_type: search_query # Or search_document, classification, clustering
          metadata:
            user_id: 'embedding-test'

Multimodal Embeddings (Vertex AI)​

TrueFoundry's gateway supports Vertex AI image and video embeddings. Promptfoo's TrueFoundry embedding adapter accepts text strings and reads the text embedding vector. Image/video payloads and their additional output vectors require a custom provider.

Tool Use and Function Calling​

TrueFoundry supports tool use and function calling, compatible with the OpenAI tools format:

promptfooconfig.yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: truefoundry:openai-main/gpt-5
    config:
      tools:
        - type: function
          function:
            name: get_weather
            description: 'Get the current weather in a given location'
            parameters:
              type: object
              properties:
                location:
                  type: string
                  description: 'The city and state, e.g. San Francisco, CA'
                unit:
                  type: string
                  enum:
                    - celsius
                    - fahrenheit
              required:
                - location
      tool_choice: auto
prompts:
  - 'What is the weather in {{location}}?'
tests:
  - vars:
      location: 'San Francisco, CA'

MCP Servers (Model Context Protocol)​

TrueFoundry supports MCP servers for tool capabilities. MCP servers provide access to integrated tools like web search, code execution, and more:

promptfooconfig.yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: truefoundry:openai-main/gpt-5
    config:
      temperature: 0.7
      mcp_servers:
        - integration_fqn: 'common-tools'
          enable_all_tools: false
          tools:
            - name: 'web_search'
            - name: 'code_executor'
      iteration_limit: 20
      metadata:
        user_id: 'mcp-test'
      loggingConfig:
        enabled: true
prompts:
  - 'Search the web for {{query}} and summarize the findings'
tests:
  - vars:
      query: 'latest AI developments 2025'

MCP Configuration Options​

  • mcp_servers: Array of MCP server configurations
    • integration_fqn: Fully qualified name of the integration (e.g., 'common-tools')
    • enable_all_tools: Whether to enable all tools in the integration (boolean)
    • tools: Array of specific tools to enable (each with a name field)
  • iteration_limit: Maximum number of iterations for tool calling (default: 20)

Available MCP Integrations​

Common integrations include:

  • common-tools: Provides web_search, code_executor, and other utilities
  • Custom integrations can be configured through your TrueFoundry account

Complete Example​

Here's an example demonstrating TrueFoundry's capabilities:

promptfooconfig.yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: TrueFoundry AI Gateway evaluation

providers:
  - id: truefoundry:openai-main/gpt-5
    label: 'GPT-5 via TrueFoundry'
    config:
      temperature: 0.7
      max_completion_tokens: 1000
      metadata:
        user_id: 'eval-user'
        environment: 'testing'
      loggingConfig:
        enabled: true
      mcp_servers:
        - integration_fqn: 'common-tools'
          enable_all_tools: false
          tools:
            - name: 'web_search'
      iteration_limit: 10

  - id: truefoundry:anthropic-main/claude-sonnet-5
    label: 'Claude Sonnet 5 via TrueFoundry'
    config:
      omitDefaults: true
      max_tokens: 1000
      metadata:
        user_id: 'eval-user'
        environment: 'testing'
      loggingConfig:
        enabled: true

prompts:
  - |
    You are a helpful assistant. Answer the following question:
    {{question}}

tests:
  - vars:
      question: 'What is the capital of France?'
    assert:
      - type: contains
        value: 'Paris'

  - vars:
      question: 'Explain quantum computing in simple terms'
    assert:
      - type: llm-rubric
        value: 'Provides a clear, simple explanation of quantum computing'

  - vars:
      question: 'Search for the latest news about AI and summarize'
    assert:
      - type: llm-rubric
        value: 'Successfully searches and summarizes recent AI news'

Observability and Monitoring​

TrueFoundry provides built-in observability features. When loggingConfig.enabled is set to true, all requests are logged and can be monitored through the TrueFoundry dashboard.

Key observability features:

  • Request and response logging
  • Performance metrics (latency, tokens used)
  • Cost tracking
  • Error monitoring
  • Custom metadata for filtering and analysis

Best Practices​

  1. Use Metadata: Add meaningful metadata to track requests by user, environment, or feature
  2. Enable Logging: Set loggingConfig.enabled: true for production monitoring
  3. Model Selection: Choose models based on your use case (speed vs. quality tradeoff)
  4. MCP Servers: Use MCP servers for enhanced capabilities like web search and code execution
  5. Cost Management: Monitor token usage through TrueFoundry's dashboard

Additional Resources​

For more information about TrueFoundry's AI Gateway, visit truefoundry.com/ai-gateway.

Last updated on by renovate[bot]