Automated indirect prompt injection against tool-calling agents

Description

The paper presents a concrete, reproducible security evaluation in which attacker-controlled instructions embedded in retrieved external content steer stateful, tool-calling LLM agents toward unauthorized actions. It adapts white-box GCG and black-box TAP to AgentDojo and evaluates single-task and task-universal attacks across 80 task pairs in four domains. The reported results show that semantic black-box optimization can discover functional prompt injections more effectively than gradient-based optimization, while attack success and transferability are strongly model-dependent.

Examples

See the primary source for the paper's evaluation and defensive reproduction context: https://arxiv.org/abs/2606.10525 (opens in a new tab)

Impact

A successful indirect injection can hijack an agent’s goal and cause unauthorized tool calls, including sending messages, modifying files, initiating transactions, or exfiltrating data. In the paper’s AgentDojo evaluation, Universal TAP reportedly reached 45.2% ASR on Qwen3-4B, while the best TAP attacks against GPT-5 reportedly reached about 4.5–4.7% ASR; universal attacks nevertheless compromised a broader set of GPT-5 task pairs under repeated attempts.

Affected Systems

• AgentDojo
• Tool-calling LLM agents that process untrusted emails, documents, web pages, files, or tool outputs
• AgentDojo Workspace suite
• AgentDojo Banking suite
• AgentDojo Travel suite
• AgentDojo Slack suite

Mitigation Steps

• Treat retrieved content and tool outputs as untrusted data; preserve provenance and separate them from privileged instructions instead of relying only on role markers in a shared prompt context.
• Require deterministic authorization and user confirmation for side-effectful or high-risk tool calls, especially external messaging, file deletion, data disclosure, and financial actions.
• Apply least privilege, execution isolation, and information-flow controls so a compromised agent cannot freely access credentials, sensitive data, or consequential tools.
• Inspect and sanitize untrusted content at tool-return and privileged-decision boundaries; do not rely only on detecting obvious override language because the paper reports that effective injections can resemble normal domain documents.
• Continuously red-team agents in sandboxed, synthetic environments using deterministic state checks, both black-box semantic search and white-box optimization where applicable, universal attacks, cross-domain tasks, and repeated attempts.
• Evaluate robustness separately for every deployed model and configuration; the paper reports substantial family-specific transfer gaps, so results on one model should not be assumed to cover another.

Evidence

• [https://arxiv.org/abs/2606.10525 (opens in a new tab); Abstract] The paper explicitly evaluates automated indirect prompt injection against LLM agents by adapting white-box GCG and black-box TAP within AgentDojo, making it a concrete security evaluation rather than a purely conceptual discussion.
• [arXiv:2606.10525v1, Sections 2.4 and 3] The threat model places attacker-controlled instructions in external content retrieved during normal operation; success is a specific unauthorized action verified by AgentDojo deterministic environment-state checks.
• [arXiv:2606.10525v1, Section 6.1] The authors report 80 task pairs: 20 per Workspace, Banking, Travel, and Slack suite, constructed from five benign user tasks and four malicious injection tasks per suite. This simulated benchmark and deterministic checks provide a safe defensive reproduction reference without using production systems.
• [arXiv:2606.10525v1, Section 6.1] The main targets are Gemma3-4B Instruct, Qwen3-4B Instruct, and GPT-5; GPT-5-mini is evaluated as TAP judge and attacker, and transfer testing additionally covers Qwen3-32B, Qwen3-235B-A22B (MoE), Qwen3-235B-A22B-Thinking (MoE), Claude Sonnet 4.5, and Gemini 2.5 Flash.
• [arXiv:2606.10525v1, Section 6.2 Overall Attack Performance] The paper reports 44.6% single-task TAP and 45.2% universal TAP ASR on Qwen3-4B, versus 23.0% and 24.1% for the corresponding GCG attacks; these are paper-reported measurements, not independently verified results.
• [arXiv:2606.10525v1, Section 6.2 Model Vulnerability] The paper reports GPT-5 ASR of 4.5% for single-task TAP and 4.7% for universal TAP, with universal TAP reaching about 30.0% Success@N versus about 14% for single-task TAP.
• [arXiv:2606.10525v1, Sections 6.1 and 6.3 GCG: Cross-Model Transfer] GCG injections optimized on Qwen3-4B were evaluated without re-optimization on seven additional models; the paper reports cross-family transfer below 2% ASR for GPT-5, GPT-5-mini, and Claude Sonnet 4.5, 7.7% for universal GCG on Gemini 2.5 Flash, and 24–36% within-family Qwen transfer.
• [arXiv:2606.10525v1, Section 6.4] The qualitative analysis reports that obvious authority-override patterns often work on Qwen3-4B but largely fail on GPT-5; remaining successful GPT-5 cases can use domain-native or contextual-prerequisite framing, supporting defenses beyond simple keyword filtering.
• [arXiv:2606.10525v1, Sections 8–9] The authors state that the study evaluates attacks rather than defenses, and recommend context isolation, mandatory verification steps, instruction hierarchy, and trust reasoning; no mitigation efficacy is independently established by this paper.
• [arXiv:2606.10525v1, Section 5.2.3 and References] The TAP adaptation is described as building on the directly linked Parley implementation; the arXiv landing page exposes PDF, HTML, and TeX source but does not identify a study-specific code repository.