Skip to main content

Assertions & metrics

Assertions are used to compare the LLM output against expected values or conditions. While assertions are not required to run an eval, they are a useful way to automate your analysis.

Different types of assertions can be used to validate the output in various ways, such as checking for equality, JSON structure, similarity, or custom functions.

In machine learning, "Accuracy" is a metric that measures the proportion of correct predictions made by a model out of the total number of predictions. With promptfoo, accuracy is defined as the proportion of prompts that produce the expected or desired output.

Using assertions

To use assertions in your test cases, add an assert property to the test case with an array of assertion objects. Each assertion object should have a type property indicating the assertion type and any additional properties required for that assertion type.

Example:

tests:
- description: 'Test if output is equal to the expected value'
vars:
example: 'Hello, World!'
assert:
- type: equals
value: 'Hello, World!'

Assertion properties

PropertyTypeRequiredDescription
typestringYesType of assertion
valuestringNoThe expected value, if applicable
thresholdnumberNoThe threshold value, applicable only to certain types such as similar, cost, javascript, python
weightnumberNoHow heavily to weigh the assertion. Defaults to 1.0
providerstringNoSome assertions (similarity, llm-rubric, model-graded-*) require an LLM provider
rubricPromptstring | string[]NoModel-graded LLM prompt
configobjectNoExternal mapping of arbitrary strings to values passed to custom javascript/python assertions
transformstringNoProcess the output before running the assertion. See Transformations
metricstringNoTag that appears in the web UI as a named metric

Grouping assertions via Assertion Sets

Assertions can be grouped together using an assert-set.

Example:

tests:
- description: 'Test that the output is cheap and fast'
vars:
example: 'Hello, World!'
assert:
- type: assert-set
assert:
- type: cost
threshold: 0.001
- type: latency
threshold: 200

In the above example if all assertions of the assert-set pass the entire assert-set passes.

There are cases where you may only need a certain number of assertions to pass. Here you can use threshold.

Example - if one of two assertions need to pass or 50%:

tests:
- description: 'Test that the output is cheap or fast'
vars:
example: 'Hello, World!'
assert:
- type: assert-set
threshold: 0.5
assert:
- type: cost
threshold: 0.001
- type: latency
threshold: 200

Assertion Set properties

PropertyTypeRequiredDescription
typestringYesMust be assert-set
assertarray of assertsYesAssertions to be run for the set
thresholdnumberNoSuccess threshold for the assert-set. Ex. 1 out of 4 equal weights assertions need to pass. Threshold should be 0.25
weightnumberNoHow heavily to weigh the assertion set within test assertions. Defaults to 1.0
metricstringNoMetric name for this assertion set within the test

Assertion types

Deterministic eval metrics

These metrics are programmatic tests that are run on LLM output. See all details

Assertion TypeReturns true if...
equalsoutput matches exactly
containsoutput contains substring
icontainsoutput contains substring, case insensitive
regexoutput matches regex
starts-withoutput starts with string
contains-anyoutput contains any of the listed substrings
contains-alloutput contains all list of substrings
icontains-anyoutput contains any of the listed substrings, case insensitive
icontains-alloutput contains all list of substrings, case insensitive
is-jsonoutput is valid json (optional json schema validation)
contains-jsonoutput contains valid json (optional json schema validation)
is-sqloutput is valid sql
contains-sqloutput contains valid sql
is-xmloutput is valid xml
contains-xmloutput contains valid xml
javascriptprovided Javascript function validates the output
pythonprovided Python function validates the output
webhookprovided webhook returns {pass: true}
rouge-nRouge-N score is above a given threshold (default 0.75)
bleuBLEU score is above a given threshold (default 0.5)
levenshteinLevenshtein distance is below a threshold
latencyLatency is below a threshold (milliseconds)
perplexityPerplexity is below a threshold
perplexity-scoreNormalized perplexity
costCost is below a threshold (for models with cost info such as GPT)
is-valid-openai-function-callEnsure that the function call matches the function's JSON schema
is-valid-openai-tools-callEnsure all tool calls match the tools JSON schema
tip

Every test type can be negated by prepending not-. For example, not-equals or not-regex.

Model-assisted eval metrics

These metrics are model-assisted, and rely on LLMs or other machine learning models.

See Model-graded evals, classification, and similarity docs for more information.

Assertion TypeMethod
similarEmbeddings and cosine similarity are above a threshold
classifierRun LLM output through a classifier
llm-rubricLLM output matches a given rubric, using a Language Model to grade output
answer-relevanceEnsure that LLM output is related to original query
context-faithfulnessEnsure that LLM output uses the context
context-recallEnsure that ground truth appears in context
context-relevanceEnsure that context is relevant to original query
factualityLLM output adheres to the given facts, using Factuality method from OpenAI eval
model-graded-closedqaLLM output adheres to given criteria, using Closed QA method from OpenAI eval
select-bestCompare multiple outputs for a test case and pick the best one

Weighted assertions

In some cases, you might want to assign different weights to your assertions depending on their importance. The weight property is a number that determines the relative importance of the assertion. The default weight is 1.

The final score of the test case is calculated as the weighted average of the scores of all assertions, where the weights are the weight values of the assertions.

Here's an example:

tests:
assert:
- type: equals
value: 'Hello world'
weight: 2
- type: contains
value: 'world'
weight: 1

In this example, the equals assertion is twice as important as the contains assertion.

If the LLM output is Goodbye world, the equals assertion fails but the contains assertion passes, and the final score is 0.33 (1/3).

Setting a score requirement

Test cases support an optional threshold property. If set, the pass/fail status of a test case is determined by whether the combined weighted score of all assertions exceeds the threshold value.

For example:

tests:
threshold: 0.5
assert:
- type: equals
value: 'Hello world'
weight: 2
- type: contains
value: 'world'
weight: 1

If the LLM outputs Goodbye world, the equals assertion fails but the contains assertion passes and the final score is 0.33. Because this is below the 0.5 threshold, the test case fails. If the threshold were lowered to 0.2, the test case would succeed.

info

If weight is set to 0, the assertion automatically passes.

Load assertions from external file

Raw files

The value of an assertion can be loaded directly from a file using the file:// syntax:

- assert:
- type: contains
value: file://gettysburg_address.txt

Javascript

If the file ends in .js, the Javascript is executed:

- assert:
- type: javascript
value: file://path/to/assert.js

The type definition is:

type AssertionValueFunctionContext = {
prompt: string | undefined;
vars: Record<string, string | object>;
test: AtomicTestCase<Record<string, string | object>>;
logProbs: number[] | undefined;
config?: Record<string, any>;
provider: ApiProvider | undefined;
providerResponse: ProviderResponse | undefined;
};
type AssertionResponse = string | boolean | number | GradingResult;
type AssertFunction = (output: string, context: AssertionValueFunctionContext) => AssertResponse;

See GradingResult definition.

Here's an example assert.js:

module.exports = (output, { vars }) => {
console.log(`Received ${output} using variables ${JSON.stringify(vars)}`);
return {
pass: true,
score: 0.5,
reason: 'Some custom reason',
};
};

You can also use Javascript files in non-javascript-type asserts. For example, using a Javascript file in a contains assertion will check that the output contains the string returned by Javascript.

Python

If the file ends in .py, the Python is executed:

- assert:
- type: python
value: file://path/to/assert.py

The assertion expects an output that is bool, float, or a JSON GradingResult.

For example:

import sys
import json

output = sys.argv[1]
context = json.loads(sys.argv[2])

# Use `output` and `context['vars']` to determine result ...

print(json.dumps({
'pass': False,
'score': 0.5,
'reason': 'Some custom reason',
}))

Load assertions from CSV

The Tests file is an optional format that lets you specify test cases outside of the main config file.

To add an assertion to a test case in a vars file, use the special __expected column.

Here's an example tests.csv:

text__expected
Hello, world!Bonjour le monde
Goodbye, everyone!fn:output.includes('Au revoir');
I am a pineapplegrade:doesn't reference any fruits besides pineapple

All assertion types can be used in __expected. The column supports exactly one assertion.

  • is-json and contains-json are supported directly, and do not require any value
  • fn indicates javascript type. For example: fn:output.includes('foo')
  • file:// indicates an external file relative to your config. For example: file://custom_assertion.py or file://customAssertion.js
  • similar takes a threshold value. For example: similar(0.8):hello world
  • grade indicates llm-rubric. For example: grade: does not mention being an AI
  • By default, __expected will use type equals

When the __expected field is provided, the success and failure statistics in the evaluation summary will be based on whether the expected criteria are met.

To run multiple assertions, use column names __expected1, __expected2, __expected3, etc.

For more advanced test cases, we recommend using a testing framework like Jest or Vitest or Mocha and using promptfoo as a library.

Reusing assertions with templates

If you have a set of common assertions that you want to apply to multiple test cases, you can create assertion templates and reuse them across your configuration.

assertionTemplates:
containsMentalHealth:
type: javascript
value: output.toLowerCase().includes('mental health')

prompts:
- file://prompt1.txt
- file://prompt2.txt
providers:
- openai:gpt-4o-mini
- localai:chat:vicuna
tests:
- vars:
input: Tell me about the benefits of exercise.
assert:
- $ref: "#/assertionTemplates/containsMentalHealth"
- vars:
input: How can I improve my well-being?
assert:
- $ref: "#/assertionTemplates/containsMentalHealth"

In this example, the containsMentalHealth assertion template is defined at the top of the configuration file and then reused in two test cases. This approach helps maintain consistency and reduces duplication in your configuration.

Defining named metrics

Each assertion supports a metric field that allows you to tag the result however you like. Use this feature to combine related assertions into aggregate metrics.

For example, these asserts will aggregate results into two metrics, Tone and Consistency.

tests:
- assert:
- type: equals
value: Yarr
metric: Tone

- assert:
- type: icontains
value: grub
metric: Tone

- assert:
- type: is-json
metric: Consistency

- assert:
- type: python
value: max(0, len(output) - 300)
metric: Consistency

- type: similar
value: Ahoy, world
metric: Tone

- assert:
- type: llm-rubric
value: Is spoken like a pirate
metric: Tone

These metrics will be shown in the UI:

llm eval metrics

See named metrics example.

Creating derived metrics

Derived metrics, also known as composite or calculated metrics, are computed at runtime based on other metrics. They are aggregated and displayed as named metrics (see above).

Derived metrics are calculated after all individual test evaluations are completed. They can be defined using mathematical expressions or custom functions that aggregate or transform the named scores collected during the tests.

Configuring derived metrics

To configure derived metrics in your test suite, you add a derivedMetrics array to the TestSuite object. Each entry in this array is an object that specifies the name of the metric and the formula or function used to calculate it.

Usage

Each derived metric has the following properties:

  • name: The name of the metric. This is used as the identifier in the output results.
  • value: The calculation method for the metric. This can be a string representing a mathematical expression or a function that takes the current scores and the evaluation context as arguments and returns a numeric value.

Example

Here's an example of how to define derived metrics in a test suite configuration:

derivedMetrics:
- name: 'EfficiencyAdjustedPerformance'
value: '(PerformanceScore / InferenceTime) * EfficiencyFactor'
# - ...

In this example, EfficiencyAdjustedPerformance is calculated using a simple mathematical expression that uses existing named scores.

info

Good to know:

  • Derived metrics are calculated in the order they are provided. You can reference previous derived metrics.
  • In order to reference a basic metric, you must name it (see named scores above).
  • In order to be used in a mathematical expression, named scores must not have any spaces or special characters in them.

Running assertions directly on outputs

If you already have LLM outputs and want to run assertions on them, the eval command supports standalone assertion files.

Put your outputs in a JSON string array, like this output.json:

["Hello world", "Greetings, planet", "Salutations, Earth"]

And create a list of assertions (asserts.yaml):

- type: icontains
value: hello

- type: javascript
value: 1 / (output.length + 1) # prefer shorter outputs

- type: model-graded-closedqa
value: ensure that the output contains a greeting

Then run the eval command:

promptfoo eval --assertions asserts.yaml --model-outputs outputs.json

Tagging outputs

Promptfoo accepts a slightly more complex JSON structure that includes an output field for the model's output and a tags field for the associated tags. These tags are shown in the web UI as a comma-separated list. It's useful if you want to keep track of certain output attributes:

[
{ "output": "Hello world", "tags": ["foo", "bar"] },
{ "output": "Greetings, planet", "tags": ["baz", "abc"] },
{ "output": "Salutations, Earth", "tags": ["def", "ghi"] }
]

Processing and formatting outputs

If you need to do any processing/formatting of outputs, use a Javascript provider, Python provider, or custom script.