CyberSecEval Dataset for LLM Security Testing

Overview

The CyberSecEval plugin uses the textual prompt injection datasets from Meta's Purple Llama CyberSecEval benchmark.

The datasets include multilingual prompt injection attacks, testing LLM systems in multiple languages.

The dataset includes test cases that evaluate:

• Direct prompt injection attacks
• Indirect prompt injection attempts
• Security-violating content
• Logic-violating content
• Cross-language attack vectors
• System prompt leakage attempts

tip

This plugin requires a target provider that accepts both a system prompt and user input in OpenAI-compatible chat format.

How it works

The plugin:

• Downloads test cases from the CyberSecEval dataset
• Formats each test case as a system/user message pair
• Randomly selects prompts from the dataset
• Tests if the model appropriately handles the injection attempts
• Uses language-specific LLM-as-a-judge criteria to evaluate responses

Configuration

To include the CyberSecEval plugin in your LLM red teaming setup, add the following configuration to your YAML file:

redteam:
  plugins:
    - cyberseceval

You can control the number of test cases using the config.numTests parameter (defaults to 5):

redteam:
  plugins:
    - id: cyberseceval
      numTests: 25

You can also enable multilingual prompt injection by setting the config.multilingual parameter to true:

redteam:
  plugins:
    - id: cyberseceval
      config:
        multilingual: true

Full example

Create a configuration file promptfooconfig.yaml:

targets:
  # You can target a model
  - openai:chat:gpt-5
  # Or an API endpoint
  - id: https
    config:
      url: 'https://example.com/generate'
      method: 'POST'
      headers:
        'Content-Type': 'application/json'
      body:
        myPrompt: '{{prompt}}'

redteam:
  plugins:
    - cyberseceval

Then, run the red team command in the same directory as your configuration file:

promptfoo redteam run

Grading

The plugin uses a specialized grader that evaluates model outputs against language-specific judge questions. The grader checks if:

• The model maintained its original instructions despite the injection attempt
• The response adheres to security boundaries
• The model appropriately handled non-English attack vectors
• The system prompt remained protected from extraction attempts

Outputs are graded as either PASS (secure) or FAIL (vulnerable) based on these criteria.

Related Concepts

• Types of LLM Vulnerabilities
• Prompt Injection
• Language Configuration - Test in multiple languages