📄️ CI/CD
When scaling an LLM app, it's essential to be able to measure the impact of any prompt or model change. This guide shows how to use integrate promptfoo with CI/CD workflows to automatically evaluate test cases and ensure quality.
📄️ CircleCI
This guide shows how to integrate promptfoo's LLM evaluation into your CircleCI pipeline. This allows you to automatically test your prompts and models whenever changes are made to your repository.
📄️ GitHub Actions
This guide describes how to automatically run a before vs. after evaluation of edited prompts using the promptfoo GitHub Action.
📄️ Google Sheets
promptfoo allows you to import eval test cases directly from Google Sheets. This can be done either unauthenticated (if the sheet is public) or authenticated using Google's Default Application Credentials, typically with a service account for programmatic access.
📄️ Helicone
Helicone is an open source observability platform that proxies your LLM requests and provides key insights into your usage, spend, latency and more.
📄️ Jest & Vitest
promptfoo can be integrated with test frameworks like Jest and Vitest to evaluate prompts as part of existing testing and CI workflows.
📄️ Langfuse
Langfuse is an AI platform that includes prompt management capabilities.
📄️ Mocha/Chai
promptfoo can be integrated with test frameworks like Mocha and assertion libraries like Chai in order to evaluate prompts as part of existing testing and CI workflows.
📄️ Portkey AI
Portkey is an AI observability suite that includes prompt management capabilities.
📄️ Python Notebook
For an example of using promptfoo in a Google Colab/Jupyter Notebook, see this notebook.