Attack Context

AI agents Research

Security research involving AI agents and autonomous systems

259 matching entries

AI agents research results

259 entries

Embodied LLM Misaligned Actions

Published 7/1/2024

Analyzed 12/29/2024

Embodied Large Language Models (LLMs) are vulnerable to manipulation via voice-based interactions, leading to the execution of harmful physical actions. Attacks exploit three vulnerabilities: (1) cascading LLM jailbreaks resulting in malicious robotic commands; (2) misalignment between linguistic outputs (verbal refusal) and physical actions (command execution); and (3) conceptual deception, where seemingly benign instructions lead to harmful outcomes due to incomplete world knowledge within…

BadRobot: Manipulating Embodied LLMs in the Physical World

Evaluated models: BERT, GPT-3.5 Turbo, GPT-4 Turbo +2 more

Source: arXiv

LLM Memory Poisoning Attack

Published 7/1/2024

Analyzed 12/28/2024

A vulnerability in Retrieval-Augmented Generation (RAG)-based Large Language Model (LLM) agents allows attackers to inject malicious demonstrations into the agent's memory or knowledge base. By crafting a carefully optimized trigger, an attacker can manipulate the agent's retrieval mechanism to preferentially retrieve these poisoned demonstrations, causing the agent to produce adversarial outputs or take malicious actions even when seemingly benign prompts are used. The attack, termed…

Agentpoison: Red-teaming llm agents via poisoning memory or knowledge bases

Evaluated models: GPT-2, GPT-3.5 Turbo, Llama 3 70B +2 more

Source: arXiv

Social Facilitation Jailbreak

Published 7/1/2024

Analyzed 1/26/2025

The SoP framework allows for automated generation of jailbreak prompts, bypassing safety mechanisms in LLMs. SoP utilizes multiple automatically optimized "jailbreak characters" within a single prompt to persuade the LLM to generate harmful or undesirable content, even without any seed jailbreak templates. This vulnerability is demonstrated against GPT-3.5, GPT-4, and LLaMA-2.

Sop: Unlock the power of social facilitation for automatic jailbreak attack

Evaluated models: GPT-3.5 Turbo, GPT-4, Llama 2 7B Chat

Source: arXiv

Model Combination Misuse

Published 6/1/2024

Analyzed 1/26/2025

LLMs, even when individually assessed as "safe," can be combined by an adversary to achieve malicious outcomes. This vulnerability exploits the complementary strengths of multiple models—a high-capability model that refuses malicious requests and a low-capability model that does not—through task decomposition. Adversaries can either manually decompose tasks into benign (solved by the high-capability model) and easily-malicious subtasks (solved by the low-capability model) or automate the…

Adversaries can misuse combinations of safe models

Evaluated models: Claude 3 Haiku, Claude 3 Opus, Claude 3 Sonnet +8 more

Source: arXiv

Multi-turn Semantic Jailbreak

Published 5/1/2024

Analyzed 3/4/2025

A vulnerability in large language models (LLMs) allows attackers to elicit unsafe or unethical responses through a chain of semantically relevant multi-turn prompts. The attack, termed "Chain of Attack" (CoA), exploits the model's contextual understanding and adaptive response capabilities to gradually steer the conversation towards the desired harmful output, even if single-turn prompts are rejected due to safety mechanisms. The attack leverages semantic similarity scoring (e.g., using…

Chain of attack: a semantic-driven contextual multi-turn attacker for llm

Evaluated models: Baichuan 2 7B Chat, ChatGLM2 6B, GPT-3.5 Turbo +2 more

Source: arXiv

Visual Modality Jailbreak

Published 5/1/2024

Analyzed 12/28/2024

A vulnerability in multimodal large language models (MLLMs) allows for efficient jailbreaking attacks by leveraging visual input to bypass safety mechanisms. The attack constructs a multimodal model by adding a visual module to the target LLM, then uses a modified PGD algorithm to optimize visual input to generate jailbreaking embeddings. These embeddings are then converted back into text and appended to harmful queries, successfully eliciting objectionable content from the target LLM.

Efficient LLM-Jailbreaking by Introducing Visual Modality

Evaluated models: ChatGLM 6B, GPT-3.5 Turbo, Mistral 7B

Source: arXiv

Multi-Turn Crescendo Jailbreak

Published 4/1/2024

Analyzed 12/28/2024

Large Language Models (LLMs) are vulnerable to the "Crescendo" multi-turn jailbreak attack. This attack uses a series of benign, escalating prompts to gradually lead the LLM into generating harmful or disallowed content, bypassing built-in safety mechanisms. The attack leverages the LLM's tendency to follow conversational patterns and build upon previous responses, making it difficult to detect based solely on individual prompts.

Great, now write an article about that: The crescendo multi-turn llm jailbreak attack

Evaluated models: Claude 2, Claude 3 Opus, Claude 3.5 Sonnet +6 more

Source: arXiv

Infectious Multi-Agent Jailbreak

Published 2/1/2024

Analyzed 12/29/2024

Multimodal Large Language Models (MLLMs) in multi-agent environments are vulnerable to "infectious jailbreak," where a single adversarial image injected into the memory of one agent can cause nearly all agents to exhibit harmful behaviors exponentially fast through agent-to-agent interaction. The adversarial image acts as a "virus," spreading via pairwise chats without further attacker intervention.

Agent smith: A single image can jailbreak one million multimodal llm agents exponentially fast

Evaluated models: GPT-4V, InstructBLIP, LLaVA 1.5

Source: arXiv

Autonomous Agent Jailbreak

Published 11/1/2023

Analyzed 12/29/2024

Large Language Model (LLM)-based agents, due to their multi-agent architecture and role-based interactions, are vulnerable to adversarial attacks that exploit the system's design and agent roles. Maliciously crafted prompts, particularly those targeting system-level roles, can cause agents to generate harmful content, bypassing safety mechanisms more effectively than attacks against individual LLMs. The vulnerability stems from a "domino effect" where one compromised agent can trigger harmful…

Evil geniuses: Delving into the safety of llm-based agents

Evaluated models: GPT-3.5 Turbo, GPT-4

Source: arXiv

Related research categories