Llama.cpp

The llama provider connects to the HTTP server bundled with llama.cpp.

Configuration

To use the llama provider, specify llama as the provider in your promptfooconfig.yaml file.

Supported environment variables:

• LLAMA_BASE_URL - Scheme, hostname, and port (defaults to http://localhost:8080)
• REQUEST_TIMEOUT_MS - Request timeout in milliseconds

note

The server address comes from LLAMA_BASE_URL only. A config.baseUrl key is ignored.

Configuration options

Promptfoo sends these options to the llama.cpp server's /completion endpoint:

Option	Type	Description
n_predict	number	Tokens to generate. Defaults to 512.
temperature	number	Sampling temperature.
top_k	number	Top-k sampling.
top_p	number	Nucleus sampling.
n_keep	number	Tokens from the prompt to retain when context fills.
stop	string[]	Sequences that stop generation.
repeat_penalty	number	Penalty applied to repeated tokens.
repeat_last_n	number	How far back to apply repeat_penalty.
penalize_nl	boolean	Whether newlines are penalized.
presence_penalty	number	Penalizes tokens by presence.
frequency_penalty	number	Penalizes tokens by frequency.
mirostat	0, 1, or 2	Mirostat sampling: 0 disables it, 1 uses Mirostat, and 2 uses Mirostat 2.0.
mirostat_tau	number	Mirostat target entropy.
mirostat_eta	number	Mirostat learning rate.
seed	number	Random seed.
ignore_eos	boolean	Continue past the end-of-sequence token.
logit_bias	array or object	Token/bias pairs or an OpenAI-style bias map. Use false in a pair to ban a token.

providers:
  - id: llama
    config:
      n_predict: 1024
      temperature: 0
      stop: ['</s>']

For a detailed example of how to use Promptfoo with llama.cpp, including configuration and setup, refer to the example on GitHub.