Impact

Integrity Research

Security research involving model-output integrity and reliability

539 matching entries

Integrity research results

539 entries

LLM Self-Jailbreaking Attack

Published 2/1/2025

Analyzed 3/4/2025

Large Language Models (LLMs) with refusal training are vulnerable to a "jailbreaking-to-jailbreak" (J2) attack. A J2 attack involves initially jailbreaking a powerful LLM to create a "J2 attacker." This attacker, instructed with general jailbreaking strategies, then autonomously attempts to jailbreak other LLMs, including potentially the same model it was derived from, by iteratively refining its attack based on previous attempts and in-context learning.

Jailbreaking to Jailbreak

Evaluated models: Claude 3.5 Haiku, Claude 3.5 Sonnet, Gemini 1.5 Pro +2 more

Source: arXiv

LLM Syntax Jailbreak

Published 2/1/2025

Analyzed 3/4/2025

Large Language Models (LLMs) are vulnerable to structure transformation attacks, where malicious prompts are encoded in diverse syntax spaces (e.g., SQL, JSON, LLM-generated syntaxes) to bypass safety mechanisms. These attacks maintain the harmful intent while altering the linguistic structure, making detection based on token-level patterns ineffective.

StructTransform: A Scalable Attack Surface for Safety-Aligned Large Language Models

Evaluated models: BERT, Claude 3.5 Sonnet, GPT-4o +5 more

Source: arXiv

LLM Watermark Neutralization

Published 2/1/2025

Analyzed 12/30/2025

Large Language Model (LLM) watermarking schemes based on n-gram probability biases (specifically KGW, SynthID-Text, MinHash, and SkipHash) are vulnerable to adversarial removal during Knowledge Distillation. When a student model is trained on the output of a watermarked teacher model, it inherits the watermark's statistical biases ("radioactivity"). An attacker can exploit this inheritance by comparing the student model's output token probabilities against a base model to extract the…

Can LLM Watermarks Robustly Prevent Unauthorized Knowledge Distillation?

Evaluated models: GLM 4 9B Chat, Llama 7B, Llama 3.2 1B

Source: arXiv

Learned Instruction Rewriting Jailbreak

Published 2/1/2025

Analyzed 3/4/2025

Large Language Models (LLMs) are vulnerable to "Rewrite to Jailbreak" (R2J) attacks. R2J exploits the models' safety mechanisms by iteratively rewriting harmful prompts, subtly altering wording to bypass safety filters while maintaining the original malicious intent. This differs from previous methods which rely on adding extraneous prefixes/suffixes or creating forced instruction-following scenarios, thus being more difficult to detect.

Rewrite to Jailbreak: Discover Learnable and Transferable Implicit Harmfulness Instruction

Evaluated models: Gemini Pro, GPT-3.5 Turbo, Llama 2 7B Chat +1 more

Source: arXiv

Multi-Dimensional Safety Bypass

Published 2/1/2025

Analyzed 4/12/2025

Large Language Models (LLMs) trained with safety fine-tuning techniques are vulnerable to multi-dimensional evasion attacks. Safety-aligned behavior, such as refusing harmful queries, is controlled not by a single direction in activation space, but by a subspace of interacting directions. Manipulating non-dominant directions, which represent distinct jailbreak patterns or indirect features, can suppress the dominant direction responsible for refusal, thereby bypassing learned safety…

The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Safety Analysis

Evaluated models: Llama 3 8B, Llama 3.1 405B Instruct, Llama 3.1 8B Instruct +2 more

Source: arXiv

Multi-Turn Foot-In-The-Door Jailbreak

Published 2/1/2025

Analyzed 3/4/2025

A multi-turn prompt injection attack, termed "Foot-In-The-Door" (FITD), exploits the psychological principle of incremental commitment to progressively escalate malicious requests, bypassing LLM safety mechanisms. The attack leverages intermediate "bridge" prompts and self-alignment techniques to coax the model into generating increasingly harmful outputs, even when initially refusing similar direct requests.

Foot-In-The-Door: A Multi-turn Jailbreak for LLMs

Evaluated models: GPT-4o, GPT-4o Mini, Llama 3 8B Instruct +4 more

Source: arXiv

Multimodal Flanking Jailbreak

Published 2/1/2025

Analyzed 3/4/2025

A novel "Flanking Attack" exploits the vulnerability of multimodal LLMs (e.g., Google Gemini) to bypass content moderation filters by embedding adversarial prompts within a sequence of benign prompts. The attack leverages the LLM's processing of both audio and text, obfuscating harmful requests through contextualization and layering, thereby yielding policy-violating responses.

From Compliance to Exploitation: Jailbreak Prompt Attacks on Multimodal LLMs

Evaluated models: Not reported

Source: arXiv

One-Shot LLM Steering Attack

Published 2/1/2025

Analyzed 3/4/2025

Large Language Models (LLMs) are vulnerable to one-shot steering vector optimization attacks. By applying gradient descent to a single training example, an attacker can generate steering vectors that induce or suppress specific behaviors across multiple inputs, even those unseen during the optimization process. This allows malicious actors to manipulate the model's output in a generalized way, bypassing safety mechanisms designed to prevent harmful responses.

Investigating Generalization of One-shot LLM Steering Vectors

Evaluated models: Gemma 2 2B, Gemma 2 2B IT, Llama 13B +2 more

Source: arXiv

Query Code Jailbreak

Published 2/1/2025

Analyzed 3/4/2025

Large Language Models (LLMs) are vulnerable to QueryAttack, a novel jailbreak technique that leverages structured, non-natural query languages (e.g., SQL, URL formats, or other programming language constructs) to bypass safety alignment mechanisms. The attack translates malicious natural language queries into these structured formats, exploiting the LLM's ability to understand and process such languages without triggering safety filters designed for natural language prompts. The LLM then…

QueryAttack: Jailbreaking Aligned Large Language Models Using Structured Non-natural Query Language

Evaluated models: DeepSeek Chat, DeepSeek R1, Gemini 1.5 Flash +11 more

Source: arXiv

Reasoning-Augmented Jailbreak

Published 2/1/2025

Analyzed 3/4/2025

Large Language Models (LLMs) are vulnerable to multi-turn jailbreak attacks leveraging the model's reasoning capabilities. The attack, RACE, reformulates harmful queries into benign reasoning tasks, exploiting the LLM's ability to perform complex reasoning to ultimately generate unsafe content. This bypasses standard safety mechanisms designed to prevent the generation of harmful responses.

Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Models

Evaluated models: DeepSeek R1, Gemini 1.5 Pro, Gemini 2.0 Flash Thinking +6 more

Source: arXiv

Related research categories