Skip to main content

GitHub Action

Automatically scan pull requests for LLM security vulnerabilities using the promptfoo Code Scan Action.

After scanning, the action posts findings with severity levels and suggested fixes as PR review comments.

Code Scan Action results on PR

Quick Start​

The easiest way to get started is by installing the Promptfoo Scanner GitHub App:

  1. Install the GitHub App: Go to github.com/apps/promptfoo-scanner and install the app
  2. Select repositories: Choose which repositories to enable scanning for
  3. Submit your email or sign in: You'll be redirected to promptfoo.dev to either submit your email or sign in to your account (an account is not required—just a valid email address)
  4. Review the setup PR: A pull request will be automatically opened in each respository you selected in step 2—it adds the Code Scan Action workflow to .github/workflows/promptfoo-code-scan.yml
  5. Merge the PR: you can tweak the workflow configuration if desired, and merge when ready.

Once merged, the scanner will automatically run on future pull requests, posting review comments for any security issues found.

info

When using the GitHub App:

  • Authentication is handled automatically with GitHub OIDC. No API key, token, or other configuration is needed.
  • No Promptfoo Cloud account is needed—just a valid email address.

Configuration​

Action Inputs​

Most CLI options from promptfoo code-scans run can be used as action inputs:

InputDescriptionDefault
api-hostPromptfoo API host URLhttps://api.promptfoo.dev
min-severityMinimum severity to report (low, medium, high, critical)medium
minimum-severityAlias for min-severitymedium
config-pathPath to .promptfoo-code-scan.yaml config fileAuto-detected
guidanceCustom guidance to tailor the scan (see [CLI docs][1])None
guidance-filePath to file containing custom guidance (see [CLI docs][1])None

[1]: More on custom guidance

Examples​

Scan with custom severity threshold:

- name: Run Promptfoo Code Scan
  uses: promptfoo/code-scan-action@v1
  with:
    min-severity: medium # Report medium, high and critical severity issues only (if omitted, all severity levels are reported)

Use custom guidance:

- name: Run Promptfoo Code Scan
  uses: promptfoo/code-scan-action@v1
  with:
    guidance: |
      Focus on the document ingestion flow.
      Treat any potential PII exposure as critical severity.

Load custom guidance from a file:

- name: Run Promptfoo Code Scan
  uses: promptfoo/code-scan-action@v1
  with:
    guidance-file: ./promptfoo-scan-guidance.md

Use config file:

- name: Run Promptfoo Code Scan
  uses: promptfoo/code-scan-action@v1
  with:
    config-path: .promptfoo-code-scan.yaml

Configuration File​

Create a .promptfoo-code-scan.yaml in your repository root. See the CLI documentation for all available options.

# Minimum severity level to report
minSeverity: medium

# Scan only PR diffs without filesystem exploration (default: false)
diffsOnly: false

# Custom guidance to tailor the scan
guidance: |
  Focus on authentication and authorization vulnerabilities.
  Treat any PII exposure as high severity.

Manual Installation​

You can also install the action manually without the GitHub App. When using manual installation:

  • Some features may not be available through the manual action installation, so the GitHub App is the recommended way to use the action
  • PR comments appear to come from the generic github-actions[bot] instead of the official Promptfoo Scanner bot with the Promptfoo logo
  • A Promptfoo Cloud account is required (rather than just a valid email address when using the GitHub App). You can sign up or sign in here.
  • You'll need a Promptfoo API token for authentication

Workflow Configuration​

Add this workflow to your repository at .github/workflows/promptfoo-code-scan.yml:

name: Promptfoo Code Scan

on:
  pull_request:
    types: [opened]

jobs:
  security-scan:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write

    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Run Promptfoo Code Scan
        uses: promptfoo/code-scan-action@v1
        env:
          PROMPTFOO_API_KEY: ${{ secrets.PROMPTFOO_API_KEY }}
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          min-severity: medium # or any other severity threshold: low, medium, high, critical
          # ... other configuration options...

See Also​