Attack Type

Jailbreak Research

Methods for bypassing model safety measures

704 matching entries

Jailbreak research results

704 entries

Humorous LLM Jailbreak

Published Apr 9, 2025Analyzed Apr 12, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to a jailbreaking attack leveraging humorous prompts. Embedding an unsafe request within a humorous context, using a fixed template, bypasses built-in safety mechanisms and elicits unsafe responses. The attack's success relies on a balance; too little or too much humor reduces effectiveness.

Bypassing Safety Guardrails in LLMs Using Humor

Evaluated models: Gemma 3 27B IT, Llama 3.1 8B Instruct, Llama 3.3 70B Instruct +1 more

Benign-Prompt Jailbreak

Published Apr 8, 2025Analyzed Apr 12, 2025Source: arXiv

Large Language Models (LLMs) exhibit Defense Threshold Decay (DTD): generating substantial benign content shifts the model's attention from the input prompt to prior outputs, increasing susceptibility to jailbreak attacks. The "Sugar-Coated Poison" (SCP) attack exploits this by first generating benign content, then transitioning to malicious output.

Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

Evaluated models: Claude 3.5 Sonnet, DeepSeek R1, GPT-3.5 Turbo +3 more

Generative Reward Hacking

Published Apr 8, 2025Analyzed Dec 9, 2025Source: arXiv

State-of-the-art Reward Models (RMs) utilized in Reinforcement Learning from Human Feedback (RLHF) exhibit poor out-of-distribution (OOD) generalization, making them susceptible to adversarial inputs. These models fail to reliably assess prompt-response pairs that diverge from their training distribution, assigning high reward scores to low-quality, nonsensical, or syntactically incorrect responses. This vulnerability allows for "reward hacking," where a policy model optimizes for unintended…

Adversarial training of reward models

Evaluated models: Llama 3.1 8B, Llama 3.3 70B, DeepSeek R1 +1 more

Multimodal Contextual Jailbreak

Published Apr 2, 2025Analyzed Apr 12, 2025Source: arXiv

Multimodal Large Language Models (MLLMs) are vulnerable to a jailbreaking attack, dubbed PiCo, that leverages token-level typographic attacks on images embedded within code-style instructions. The attack bypasses multi-tiered defense mechanisms, including input filtering and runtime monitoring, by exploiting weaknesses in the visual modality's integration with programming contexts. Harmful intent is concealed within visually benign image fragments and code instructions, circumventing safety…

PiCo: Jailbreaking Multimodal Large Language Models via Pictorial Code Contextualization

Evaluated models: Gemini 1.0 Pro Vision, GPT-4 Turbo, GPT-4o +2 more

Multi-Accent Audio Jailbreak

Published Apr 1, 2025Analyzed Apr 12, 2025Source: arXiv

Multilingual and multi-accent audio inputs, combined with acoustic adversarial perturbations (reverberation, echo, whisper effects), can bypass safety mechanisms in Large Audio Language Models (LALMs), causing them to generate unsafe or harmful outputs. The vulnerability is amplified by the interaction between acoustic and linguistic variations, particularly in languages with less training data.

Multilingual and Multi-Accent Jailbreaking of Audio LLMs

Evaluated models: DIVA Llama 3 v0 8B, MERaLion AudioLLM, MiniCPM-o 2.6 +2 more

Multi-Agent Prompt Permutation Attack

Published Mar 31, 2025Analyzed Apr 12, 2025Source: arXiv

A vulnerability in multi-agent Large Language Model (LLM) systems allows for a permutation-invariant adversarial prompt attack. By strategically partitioning adversarial prompts and routing them through a network topology, an attacker can bypass distributed safety mechanisms, even those with token bandwidth limitations and asynchronous message delivery. The attack optimizes prompt propagation as a maximum-flow minimum-cost problem, maximizing success while minimizing detection.

Agents Under Siege: Breaking Pragmatic Multi-Agent LLM Systems with Optimized Prompt Attacks

Evaluated models: DeepSeek R1 Distill, Gemma 2 9B, Llama 2 7B +7 more

Schema-Guided LLM Jailbreak

Published Mar 31, 2025Analyzed Apr 12, 2025Source: arXiv

Large Language Models (LLMs) with structured output APIs (e.g., using JSON Schema) are vulnerable to Constrained Decoding Attacks (CDAs). CDAs exploit the control plane of the LLM's decoding process by embedding malicious intent within the schema-level grammar rules, bypassing safety mechanisms that primarily focus on input prompts. The attack manipulates the allowed output space, forcing the LLM to generate harmful content despite a benign input prompt. One instance of a CDA is the Chain Enum…

Output Constraints as Attack Surface: Exploiting Structured Generation to Bypass LLM Safety Mechanisms

Evaluated models: Gemini 2.0 Flash, Gemma 2 9B, GPT-4o +5 more

Segmented Prompt Jailbreak

Published Mar 27, 2025Analyzed Apr 3, 2025Source: arXiv

Large Language Models (LLMs) incorporating safety filters are vulnerable to a "Prompt, Divide, and Conquer" attack. This attack segments a malicious prompt into smaller, seemingly benign parts, processes these segments in parallel across multiple LLMs, and then reassembles the results to generate malicious code, bypassing the safety filters. The attack's success relies on the iterative refinement of initially abstract function descriptions into concrete implementations. Individual LLM safety…

Prompt, Divide, and Conquer: Bypassing Large Language Model Safety Filters via Segmented and Distributed Prompt Processing

Evaluated models: Claude 3.5 Haiku, Claude 3.5 Sonnet, Gemini 1.5 Pro +3 more

Multimodal Narrative Jailbreak

Published Mar 24, 2025Analyzed Apr 21, 2025Source: arXiv

Multimodal Large Language Models (MLLMs) are vulnerable to a novel attack vector leveraging narrative-driven visual storytelling and role immersion to circumvent built-in safety mechanisms. The attack, termed MIRAGE, decomposes harmful queries into environment, character, and activity triplets, generating a sequence of images and text prompts that guide the MLLM through a deceptive narrative, ultimately eliciting harmful responses. The attack successfully exploits the MLLM's cross-modal…

MIRAGE: Multimodal Immersive Reasoning and Guided Exploration for Red-Team Jailbreak Attacks

Evaluated models: Gemini 1.5 Pro, GPT-4V, Grok 2 Vision +3 more

Implicit Prompt Code Jailbreak

Published Mar 23, 2025Analyzed Apr 3, 2025Source: arXiv

Large Language Models (LLMs) used for code generation are vulnerable to a jailbreaking attack that leverages implicit malicious prompts. The attack exploits the fact that existing safety mechanisms primarily rely on explicit malicious intent within the prompt instructions. By embedding malicious intent implicitly within a benign-appearing commit message accompanying a code request (e.g., in a simulated software evolution scenario), the attacker can bypass the LLM's safety filters and induce…

Smoke and Mirrors: Jailbreaking LLM-based Code Generation via Implicit Malicious Prompts

Evaluated models: Claude 3.5 Sonnet, Code Llama 13B Instruct, DeepSeek Coder 7B +3 more

Related research categories