A recent cyber espionage campaign revealed how state actors weaponized Anthropic's Claude Code - not through traditional hacking, but by convincing the AI itself to carry out malicious operations.
In this post, we reproduce the attack on Claude Code and jailbreak it to carry out nefarious deeds. We'll also show how to configure the same attack on any other agent.
As AI agents become more capable, risk increases commensurately. Simon Willison, an AI security researcher, warns of a lethal trifecta of capabilities that, when combined, open AI systems to severe exploits.
If you're building or using AI agents that handle sensitive data, you need to understand this trifecta and test your models for these vulnerabilities.
In this post, we'll explain what the lethal trifecta is and show practical steps to use Promptfoo for detecting these security holes.
If you're looking into red teaming AI systems for the first time and don't have context for red teaming, here's something I wrote for you.
The rush to integrate large language models (LLMs) into production applications has opened up a whole new world of security challenges. AI systems face unique vulnerabilities like prompt injections, data leakage, and model misconfigurations that traditional security tools just weren't built to handle.
Input manipulation techniques like prompt injections and base64-encoded attacks can dramatically influence how AI systems behave. While established security tooling gives us some baseline protection through decades of hardening, AI systems need specialized approaches to vulnerability management. The problem is, despite growing demand, relatively few organizations make comprehensive AI security tools available as open source.
If we want cybersecurity practices to take more of a foothold, particularly now that AI systems are becoming increasingly common, it's important to make them affordable and easy to use. Tools that sound intimidating and aren't intuitive will be less likely to change the culture surrounding cybersecurity-as-an-afterthought.
I spend a lot of time thinking about what makes AI red teaming software good at what it does. Feel free to skip ahead to the tool comparisons if you already know this stuff.
Is this your first foray into AI red teaming? And probably red teaming in general? Great. This is for you.
Red teaming is the process of simulating real-world attacks to identify vulnerabilities.
AI red teaming is the process of simulating real-world attacks to identify vulnerabilities in artificial-intelligence systems. There are two scopes people often use to refer to AI red teaming:
As enterprises deploy AI applications at scale, red teaming has become essential for identifying vulnerabilities before they reach production. Two prominent open-source tools have emerged in this space: Promptfoo and Microsoft's PyRIT.
| Feature | Promptfoo | PyRIT |
|---|---|---|
| Setup Time | Minutes (Web/CLI wizard) | Hours (Python scripting) |
| Attack Generation | Automatic, context-aware | Manual configuration |
| RAG Testing | Pre-built tests | Manual configuration |
| Agent Security | RBAC, tool misuse tests included | Manual configuration |
| CI/CD Integration | Built-in | Requires custom code |
| Reporting | Visual dashboards, OWASP mapping | Raw outputs |
| Learning Curve | Low | High |
| Best For | Continuous security testing | Custom deep-dives |
As LLM applications move into production, security teams face a critical challenge: how do you systematically identify vulnerabilities before attackers do?
Two open‑source tools have emerged as popular choices for LLM red teaming, each taking a fundamentally different approach to the problem.
Google's Gemini represents a significant advancement in multimodal AI, with models featuring reasoning, huge token contexts, and lightning-fast inference.
But with these powerful capabilities come unique security challenges. This guide shows you how to use Promptfoo to systematically test Gemini models for vulnerabilities through adversarial red teaming.
Gemini's multimodal processing, extended context windows, and thinking capabilities make it particularly important to test comprehensively before production deployment.
You can also jump directly to the Gemini 2.5 Pro security report and compare it to other models.
Early red teaming tools and research began with jailbreaks like "Ignore all previous instructions" and static lists of harmful prompts. At Promptfoo, we took those ideas a step further by dynamically generating attacks based on the context of the target application.
OpenAI's GPT-4.1 and GPT-4.5 represents a significant leap in AI capabilities, especially for coding and instruction following. But with great power comes great responsibility. This guide shows you how to use Promptfoo to systematically test these models for vulnerabilities through adversarial red teaming.
GPT's enhanced instruction following and long-context capabilities make it particularly interesting to red team, as these features can be both strengths and potential attack vectors.
You can also jump directly to the GPT 4.1 security report and compare it to other models.
Anthropic's Claude 4 represents a major leap in AI capabilities, especially with its extended thinking feature. But before deploying it in production, you need to test it for security vulnerabilities.
This guide shows you how to quickly red team Claude 4 Sonnet using Promptfoo, an open-source tool for adversarial AI testing.