Impact

Integrity Research

Security research involving model-output integrity and reliability

539 matching entries

Integrity research results

539 entries

Multi-Turn Foot-In-The-Door Jailbreak

Published Feb 27, 2025Analyzed Mar 4, 2025Source: arXiv

A multi-turn prompt injection attack, termed "Foot-In-The-Door" (FITD), exploits the psychological principle of incremental commitment to progressively escalate malicious requests, bypassing LLM safety mechanisms. The attack leverages intermediate "bridge" prompts and self-alignment techniques to coax the model into generating increasingly harmful outputs, even when initially refusing similar direct requests.

Foot-In-The-Door: A Multi-turn Jailbreak for LLMs

Evaluated models: GPT-4o, GPT-4o Mini, Llama 3 8B Instruct+4 more

GPT-4o
GPT-4o Mini
Llama 3 8B Instruct
Llama 3.1 8B Instruct
Mistral 7B Instruct v0.2
Qwen 1.5 7B Chat
Qwen 2 7B Instruct

One-Shot LLM Steering Attack

Published Feb 26, 2025Analyzed Mar 4, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to one-shot steering vector optimization attacks. By applying gradient descent to a single training example, an attacker can generate steering vectors that induce or suppress specific behaviors across multiple inputs, even those unseen during the optimization process. This allows malicious actors to manipulate the model's output in a generalized way, bypassing safety mechanisms designed to prevent harmful responses.

Investigating Generalization of One-shot LLM Steering Vectors

Evaluated models: Gemma 2 2B, Gemma 2 2B IT, Llama 13B+2 more

Gemma 2 2B
Gemma 2 2B IT
Llama 13B
Llama 2 13B
Llama 3.1 8B Instruct

Improved LLM Jailbreak Transferability

Published Feb 25, 2025Analyzed Mar 19, 2025Source: arXiv

Large Language Models (LLMs) employing gradient-based optimization for jailbreaking defense are vulnerable to enhanced transferability attacks due to superfluous constraints in their objective functions. Specifically, the "response pattern constraint" (forcing a specific initial response phrase) and the "token tail constraint" (penalizing variations in the response beyond a fixed prefix) limit the search space and reduce the effectiveness of attacks across different models. Removing these…

Guiding not Forcing: Enhancing the Transferability of Jailbreaking Attacks on LLMs via Removing Superfluous Constraints

Evaluated models: Gemma 7B IT, GPT-3.5 Turbo, GPT-4 Turbo+5 more

Gemma 7B IT
GPT-3.5 Turbo
GPT-4 Turbo
Llama 2 7B Chat
Llama 3 8B Instruct
Qwen 2 7B
Vicuna 7B v1.5
Yi 1.5 9B Chat

Metaphor-Based LLM Jailbreak

Published Feb 25, 2025Analyzed Mar 19, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to a novel jailbreaking attack leveraging adversarial metaphors. The attack, termed AVATAR, induces the LLM to reason about benign metaphors related to harmful tasks, ultimately leading to the generation of harmful content either directly or through calibration of metaphorical and professional harmful content. The attack exploits the LLM's cognitive mapping process, bypassing standard safety mechanisms.

from Benign import Toxic: Jailbreaking the Language Model via Adversarial Metaphors

Evaluated models: Claude 3.5 Sonnet, GLM 3 6B, GPT-4o+8 more

Claude 3.5 Sonnet
GLM 3 6B
GPT-4o
GPT-4o Mini
Llama 3 8B
Llama 3.1 8B
Mistral 7B
o1
Qwen 2 72B
Qwen 2.5 32B
Qwen 2.5 7B

Zero-Perturbation Emoji Attack

Published Feb 24, 2025Analyzed Dec 30, 2025Source: arXiv

The Emoti-Attack vulnerability constitutes a zero-word-perturbation adversarial attack against Natural Language Processing (NLP) systems and Large Language Models (LLMs). The vulnerability exploits the discrete embedding space of emojis and emoticons to manipulate model behavior without altering the semantic content or character integrity of the original text. By appending strategically optimized emoji sequences to the prefix and suffix of an input string (formalized as $s \oplus x \oplus…

Emoti-Attack: Zero-Perturbation Adversarial Attacks on NLP Systems via Emoji Sequences

Evaluated models: Qwen 2.5 7B Instruct, Llama 3 8B Instruct, GPT-4o+4 more

Qwen 2.5 7B Instruct
Llama 3 8B Instruct
GPT-4o
Claude 3.5 Sonnet
Gemini Exp 1206
BERT
RoBERTa

Attention-Based Jailbreak

Published Feb 21, 2025Analyzed Mar 4, 2025Source: arXiv

A vulnerability in large language models (LLMs) allows attackers to bypass safety-alignment mechanisms by manipulating the model's internal attention weights. The attack, termed "Attention Eclipse," modifies the attention scores between specific tokens within a prompt, either amplifying or suppressing attention to selectively strengthen or weaken the influence of certain parts of the prompt on the model's output. This allows injection of malicious content while appearing benign to the model's…

Attention Eclipse: Manipulating Attention to Bypass LLM Safety-Alignment

Evaluated models: GPT-3.5 Turbo, GPT-4o Mini, Llama 2 13B Chat+3 more

GPT-3.5 Turbo
GPT-4o Mini
Llama 2 13B Chat
Llama 2 70B Chat
Llama 2 7B Chat
Vicuna 13B

Agent-in-the-Middle Attack

Published Feb 20, 2025Analyzed Mar 4, 2025Source: arXiv

A vulnerability exists in the communication mechanisms of Large Language Model (LLM)-based Multi-Agent Systems (LLM-MAS) enabling an Agent-in-the-Middle (AiTM) attack. An attacker can intercept and manipulate messages between agents, causing the victim agent to produce malicious outputs. The attack does not require compromising individual agents directly; instead, it leverages contextual manipulation of inter-agent communications.

Red-Teaming LLM Multi-Agent Systems via Communication Attacks

Evaluated models: GPT-3.5 Turbo, GPT-4o

ICL Permutation Exploit

Published Feb 20, 2025Analyzed Jan 14, 2026Source: arXiv

Autoregressive Large Language Models (LLMs) utilizing In-Context Learning (ICL) are vulnerable to demonstration permutation attacks due to inherent sensitivity to the ordering of input examples. This vulnerability arises from the limitations of unidirectional attention mechanisms and standard Empirical Risk Minimization (ERM) training, which fails to account for worst-case input permutations. An attacker can exploit this by permuting the order of valid, semantically correct few-shot…

PEARL: Towards permutation-resilient LLMs

Evaluated models: Llama 2 7B, Llama 3 8B, Mistral 7B+1 more

Llama 2 7B
Llama 3 8B
Mistral 7B
Gemma 7B

Iterative Chaos Jailbreak

Published Feb 19, 2025Analyzed Mar 4, 2025Source: arXiv

Large Reasoning Models (LRMs) are vulnerable to a novel jailbreak attack, "Mousetrap," which leverages the models' reasoning capabilities to elicit harmful responses. Mousetrap uses a "Chaos Machine" to iteratively transform prompts via one-to-one mappings (e.g., character substitutions, word reversals), creating complex reasoning chains that confuse the LRM and cause it to generate unsafe outputs despite safety mechanisms. The iterative nature of the attack, combined with role-playing…

A Mousetrap: Fooling Large Reasoning Models for Jailbreak with Chain of Iterative Chaos

Evaluated models: Claude 3.5 Sonnet, Gemini 2.0 Flash Thinking, o1-mini

Context-Coherent LLM Jailbreak

Published Feb 17, 2025Analyzed Mar 4, 2025Source: arXiv

A context-coherent jailbreak attack (CCJA) allows bypassing safety mechanisms in aligned large language models (LLMs) by optimizing perturbations in the continuous word embedding space of a masked language model (MLM). The attack leverages the MLM's ability to reconstruct text from hidden states to generate semantically coherent yet malicious prompts that induce the target LLM to produce unsafe outputs, even with strong safety alignment. The attack's effectiveness is enhanced by using a seed…

CCJA: Context-Coherent Jailbreak Attack for Aligned Large Language Models

Evaluated models: Guanaco 13B, Llama 2 7B Chat, Llama 3 8B Instruct+4 more

Guanaco 13B
Llama 2 7B Chat
Llama 3 8B Instruct
Mistral 7B Instruct v0.2
Mistral 7B Instruct v0.3
Vicuna 13B v1.5
Vicuna 7B v1.5

Related research categories