Attack Context

AI agents Research

Security research involving AI agents and autonomous systems

267 matching entries

AI agents research results

267 entries

LLM Agent Automates Backdoor Injection

Published 11/1/2025

Analyzed 12/1/2025

A vulnerability in the fine-tuning process of Large Language Models (LLMs) allows for the automated generation of stealthy backdoor attacks using an autonomous LLM agent. This method, termed AutoBackdoor, creates a pipeline to generate semantically coherent trigger phrases and corresponding poisoned instruction-response pairs. Unlike traditional backdoor attacks that rely on fixed, often anomalous triggers, this technique produces natural language triggers that are contextually relevant and…

AutoBackdoor: Automating Backdoor Attacks via LLM Agents

Evaluated models: GPT-4o, GPT-4o Mini, Llama 3.1 8B Instruct +3 more

Source: arXiv

LLM App Malicious Drift

Published 11/1/2025

Analyzed 12/8/2025

Improper restriction of the "Capability Space" in Large Language Model (LLM) applications allows remote attackers to manipulate application behavior through "Goal Deviation" attacks. This vulnerability arises when developers rely on the broad capabilities of a foundational model (e.g., GPT-4, LLaMA) without implementing sufficient negative constraints or disabling default plugins (e.g., DALL-E, Web Search) in the system prompt. Attackers can exploit this via natural language inputs to trigger…

Beyond Jailbreak: Unveiling Risks in LLM Applications Arising from Blurred Capability Boundaries

Evaluated models: Not reported

Source: arXiv

MAS Pipeline Weak Links

Published 11/1/2025

Analyzed 1/14/2026

Centralized Multi-Agent Systems (MAS) architectures, specifically those implemented using frameworks such as Magentic-One, LangGraph, and OpenAI Agents, contain design-level vulnerabilities facilitating safety alignment bypasses. Two primary mechanisms drive this vulnerability: 1. Context Fragmentation via Atomic Delegation: The orchestrator decomposes adversarial prompts into atomic, granular sub-tasks (e.g., "click button," "write function header") before delegating them to sub-agents. These…

Exposing Weak Links in Multi-Agent Systems under Adversarial Prompting

Evaluated models: GPT-4o

Source: arXiv

Multi-Agent Typo Vulnerability

Published 11/1/2025

Analyzed 12/30/2025

Multi-agent Large Language Model (LLM) systems employing ensemble sampling-and-voting strategies (specifically the "Agent Forest" framework) are vulnerable to adversarial input perturbations. While increasing the number of agents ($n \in \{1, \dots, 25\}$) improves accuracy on clean inputs, the system fails to mitigate the impact of synthetic punctuation noise and human-like typographical errors. Attackers can introduce surface-level perturbations—such as random punctuation insertion (10-50%…

More Agents Improve Math Problem Solving but Adversarial Robustness Gap Persists

Evaluated models: Llama 3.1 8B, Mistral 7B, Qwen 3 4B +3 more

Source: arXiv

Needle-in-Haystack Jailbreak

Published 11/1/2025

Analyzed 12/8/2025

A safety bypass vulnerability, dubbed "Ninja" (Needle-in-a-haystack jailbreak), exists in long-context Large Language Models (LLMs). The vulnerability exploits a degradation in safety alignment that occurs when a harmful goal is embedded within a massive, benign context window. Unlike traditional adversarial attacks that use unintelligible strings or "many-shot" attacks that use harmful examples, this method utilizes thematically relevant but innocuous text (the "haystack"). The attack…

Jailbreaking in the Haystack

Evaluated models: GPT-4o, Llama 3.1 8B Instruct, Gemini 2.0 Flash +2 more

Source: arXiv

Template and Suffix Optimization

Published 11/1/2025

Analyzed 12/1/2025

A vulnerability exists in multiple Large Language Models (LLMs) that allows for safety alignment bypass through an advanced jailbreaking technique called Template and Suffix Optimization (TASO). The attack combines two distinct optimization methods in an alternating, iterative feedback loop. First, a semantically meaningless adversarial suffix is optimized (e.g., using gradient-based methods like GCG) to force the LLM to begin its response with an affirmative phrase (e.g., "Sure, here is...")…

TASO: Jailbreak LLMs via Alternative Template and Suffix Optimization

Evaluated models: Baichuan 2 13B, Baichuan 2 7B, DeepSeek 7B +27 more

Source: arXiv

AI Browser Indirect Injection

Published 10/1/2025

Analyzed 12/30/2025

Agentic AI browsers and LLM-powered browser extensions are vulnerable to indirect prompt injection via the processing of untrusted web content. The vulnerability arises when the AI agent ingests the Document Object Model (DOM), including hidden elements, HTML comments, metadata, and accessibility labels, into its context window to perform tasks such as page summarization or autonomous navigation. Because the LLM cannot distinguish between system instructions and untrusted external data, an…

In-browser llm-guided fuzzing for real-time prompt injection testing in agentic AI browsers

Evaluated models: GPT-4, Llama 3.1 70B, Llama 3.3 70B

Source: arXiv

Adaptive Monitor Injection

Published 10/1/2025

Analyzed 12/9/2025

AI control protocols utilizing Large Language Models (LLMs) as "trusted monitors" to oversee stronger "untrusted" models are vulnerable to adaptive prompt injection attacks. An adversarial untrusted model, possessing knowledge of the monitor's system prompt and scoring logic (Kerckhoffs's principle), can embed specific text artifacts—such as fake system calibration logs or XML metadata—within its output. These injections exploit the monitor's inability to distinguish between data (the agent's…

Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols

Evaluated models: GPT-4o, GPT-4o Mini, GPT-4.1 +3 more

Source: arXiv

Adaptive Typographic Image Injection

Published 10/1/2025

Analyzed 12/30/2025

Multimodal agents built on Large Vision-Language Models (LVLMs) are vulnerable to adaptive typographic prompt injection attacks (AgentTypo). This vulnerability allows an attacker to execute indirect prompt injection by embedding adversarial text prompts directly into images (e.g., webpage screenshots, product photos) processed by the agent. Unlike standard visual adversarial attacks that rely on noise perturbation, this method utilizes the AgentTypo framework to perform black-box Bayesian…

AgentTypo: Adaptive Typographic Prompt Injection Attacks against Black-box Multimodal Agents

Evaluated models: GPT-4o, GPT-4V, GPT-4o Mini +2 more

Source: arXiv

Agent Harassment Escalation

Published 10/1/2025

Analyzed 2/21/2026

Large Language Model (LLM) agents powered by LLaMA-3.1-8B-Instruct and Gemini-2.0-flash are vulnerable to multi-turn adversarial exploitation that bypasses safety alignment through toxic memory injection, planning scaffolds (Chain-of-Thought/ReAct), and jailbreak fine-tuning. Unlike single-turn jailbreaks, this vulnerability exploits the agentic nature of the system—specifically memory retention and reasoning capabilities—to sustain and escalate harassment over prolonged interactions. When…

Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attacks

Evaluated models: Llama 3.1 8B Instruct, Gemini 2.0 Flash 001

Source: arXiv

Related research categories