Skip to main content

Deterministic metrics

These metrics are created by logical tests that are run on LLM output.

Assertion TypeReturns true if...
containsoutput contains substring
contains-alloutput contains all list of substrings
contains-anyoutput contains any of the listed substrings
contains-jsonoutput contains valid json (optional json schema validation)
costInference cost is below a threshold
equalsoutput matches exactly
icontainsoutput contains substring, case insensitive
icontains-alloutput contains all list of substrings, case insensitive
icontains-anyoutput contains any of the listed substrings, case insensitive
is-jsonoutput is valid json (optional json schema validation)
is-valid-openai-function-callEnsure that the function call matches the function's JSON schema
is-valid-openai-tools-callEnsure all tool calls match the tools JSON schema
javascriptprovided Javascript function validates the output
latencyLatency is below a threshold (milliseconds)
levenshteinLevenshtein distance is below a threshold
perplexity-scoreNormalized perplexity
perplexityPerplexity is below a threshold
pythonprovided Python function validates the output
regexoutput matches regex
rouge-nRouge-N score is above a given threshold
starts-withoutput starts with string
webhookprovided webhook returns {pass: true}
tip

Every test type can be negated by prepending not-. For example, not-equals or not-regex.

Assertion types

Contains

The contains assertion checks if the LLM output contains the expected value.

Example:

assert:
- type: contains
value: 'The expected substring'

The icontains is the same, except it ignores case:

assert:
- type: icontains
value: 'The expected substring'

Contains-All

The contains-all assertion checks if the LLM output contains all of the specified values.

Example:

assert:
- type: contains-all
value:
- 'Value 1'
- 'Value 2'
- 'Value 3'

Contains-Any

The contains-any assertion checks if the LLM output contains at least one of the specified values.

Example:

assert:
- type: contains-any
value:
- 'Value 1'
- 'Value 2'
- 'Value 3'

For case insensitive matching, use icontains-any.

For case insensitive matching, use icontains-all.

Regex

The regex assertion checks if the LLM output matches the provided regular expression.

Example:

assert:
- type: regex
value: "\\d{4}" # Matches a 4-digit number

Contains-JSON

The contains-json assertion checks if the LLM output contains a valid JSON structure.

Example:

assert:
- type: contains-json

You may optionally set a value as a JSON schema in order to validate the JSON contents:

assert:
- type: contains-json
value:
required: [latitude, longitude]
type: object
properties:
latitude:
minimum: -90
type: number
maximum: 90
longitude:
minimum: -180
type: number
maximum: 180

JSON is valid YAML, so you can also just copy in any JSON schema directly:

assert:
- type: contains-json
value:
{
'required': ['latitude', 'longitude'],
'type': 'object',
'properties':
{
'latitude': { 'type': 'number', 'minimum': -90, 'maximum': 90 },
'longitude': { 'type': 'number', 'minimum': -180, 'maximum': 180 },
},
}

If your JSON schema is large, import it from a file:

assert:
- type: contains-json
value: file://./path/to/schema.json

See also: is-json

Cost

The cost assertion checks if the cost of the LLM call is below a specified threshold.

This requires LLM providers to return cost information. Currently this is only supported by OpenAI GPT models and custom providers.

Example:

providers:
- openai:gpt-3.5-turbo
- openai:gpt-4
assert:
# Pass if the LLM call costs less than $0.001
- type: cost
threshold: 0.001

Equality

The equals assertion checks if the LLM output is equal to the expected value.

Example:

assert:
- type: equals
value: 'The expected output'

You can also check whether it matches the expected JSON format.

assert:
- type: equals
value: { 'key': 'value' }

If your expected JSON is large, import it from a file:

assert:
- type: equals
value: 'file://path/to/expected.json'

Is-JSON

The is-json assertion checks if the LLM output is a valid JSON string.

Example:

assert:
- type: is-json

You may optionally set a value as a JSON schema. If set, the output will be validated against this schema:

assert:
- type: is-json
value:
required: [latitude, longitude]
type: object
properties:
latitude:
minimum: -90
type: number
maximum: 90
longitude:
minimum: -180
type: number
maximum: 180

JSON is valid YAML, so you can also just copy in any JSON schema directly:

assert:
- type: is-json
value:
{
'required': ['latitude', 'longitude'],
'type': 'object',
'properties':
{
'latitude': { 'type': 'number', 'minimum': -90, 'maximum': 90 },
'longitude': { 'type': 'number', 'minimum': -180, 'maximum': 180 },
},
}

If your JSON schema is large, import it from a file:

assert:
- type: is-json
value: file://./path/to/schema.json

is-valid-openai-function-call

This ensures that any JSON LLM output adheres to the schema specified in the functions configuration of the provider. Learn more about the OpenAI provider.

is-valid-openai-tools-call

This ensures that any JSON LLM output adheres to the schema specified in the tools configuration of the provider. Learn more about the OpenAI provider.

Javascript

See Javascript assertions.

Latency

The latency assertion passes if the LLM call takes longer than the specified threshold. Duration is specified in milliseconds.

Example:

assert:
# Fail if the LLM call takes longer than 5 seconds
- type: latency
threshold: 5000

Note that latency requires that the cache is disabled with promptfoo eval --no-cache or an equivalent option.

Levenshtein distance

The levenshtein assertion checks if the LLM output is within a given edit distance from an expected value.

Example:

assert:
# Ensure Levenshtein distance from "hello world" is <= 5
- type: levenshtein
threshold: 5
value: hello world

Perplexity

Perplexity is a measurement used in natural language processing to quantify how well a language model predicts a sample of text. It's essentially a measure of the model's uncertainty.

High perplexity suggests it is less certain about its predictions, often because the text is very diverse or the model is not well-tuned to the task at hand.

Low perplexity means the model predicts the text with greater confidence, implying it's better at understanding and generating text similar to its training data.

To specify a perplexity threshold, use the perplexity assertion type:

assert:
# Fail if the LLM is below perplexity threshold
- type: perplexity
threshold: 1.5
warning

Perplexity requires the LLM API to output logprobs. Currently only more recent versions of OpenAI GPT and Azure OpenAI GPT APIs support this.

Comparing different outputs from the same LLM

You can compare perplexity scores across different outputs from the same model to get a sense of which output the model finds more likely (or less surprising). This is a good way to tune your prompts and hyperparameters (like temperature) to be more accurate.

Comparing outputs from different LLMs

Comparing scores across models may not be meaningful, unless the models have been trained on similar datasets, the tokenization process is consistent between models, and the vocabulary of the models is roughly the same.

perplexity-score

perplexity-score is a supported metric similar to perplexity, except it is normalized between 0 and 1 and inverted, meaning larger numbers are better.

This makes it easier to include in an aggregate promptfoo score, as higher scores are usually better. In this example, we compare perplexity across multiple GPTs:

providers: [gpt-4-1106-preview, gpt-3.5-turbo-1106]
tests:
- assert:
- type: perplexity-score
threshold: 0.5 # optional
# ...

Python

See Python assertions.

Starts-With

The starts-with assertion checks if the LLM output begins with the specified string.

This example checks if the output starts with "Yes":

assert:
- type: starts-with
value: 'Yes'

Webhook

The webhook assertion sends the LLM output to a specified webhook URL for custom validation. The webhook should return a JSON object with a pass property set to true or false.

Example:

assert:
- type: webhook
value: 'https://example.com/webhook'

The webhook will receive a POST request with a JSON payload containing the LLM output and the context (test case variables). For example, if the LLM output is "Hello, World!" and the test case has a variable example set to "Example text", the payload will look like:

{
"output": "Hello, World!",
"context": {
"prompt": "Greet the user",
"vars": {
"example": "Example text"
}
}
}

The webhook should process the request and return a JSON response with a pass property set to true or false, indicating whether the LLM output meets the custom validation criteria. Optionally, the webhook can also provide a reason property to describe why the output passed or failed the assertion.

Example response:

{
"pass": true,
"reason": "The output meets the custom validation criteria"
}

If the webhook returns a pass value of true, the assertion will be considered successful. If it returns false, the assertion will fail, and the provided reason will be used to describe the failure.

You may also return a score:

{
"pass": true,
"score": 0.5,
"reason": "The output meets the custom validation criteria"
}