Attack Type

Jailbreak Research

Methods for bypassing model safety measures

704 matching entries

Jailbreak research results

704 entries

Shuffle Inconsistency Jailbreak

Published Jan 9, 2025Analyzed Jan 26, 2025Source: arXiv

Multimodal Large Language Models (MLLMs) exhibit a vulnerability where shuffling the order of words in text prompts or patches in image prompts can bypass their safety mechanisms, despite the model still understanding the intent of the shuffled input. This "Shuffle Inconsistency" allows attackers to elicit harmful responses by submitting shuffled harmful prompts that would otherwise be blocked.

Jailbreaking Multimodal Large Language Models via Shuffle Inconsistency

Evaluated models: Claude 3.5 Sonnet, Gemini 1.5 Pro, GPT-4o +6 more

Cybersecurity Obfuscation Jailbreak

Published Jan 2, 2025Analyzed Dec 8, 2025Source: arXiv

A multi-step prompt injection vulnerability allows attackers to bypass Large Language Model (LLM) safety guardrails by combining prompt obfuscation with task decomposition. The attack methodology, identified as part of the CySecBench research, employs a "Word Reversal" technique where every fifth word in the malicious input is reversed to evade initial keyword detection. This obfuscated input is then embedded within a benign educational context, specifically instructing the model to act as a…

CySecBench: Generative AI-based CyberSecurity-focused Prompt Dataset for Benchmarking Large Language Models

Evaluated models: Not reported

Evolutionary LLM Jailbreak

Published Dec 28, 2024Analyzed Jan 26, 2025Source: arXiv

This vulnerability allows an attacker to bypass the safety mechanisms of Large Language Models (LLMs) by using an evolutionary algorithm to generate effective jailbreak prompts. The algorithm leverages the LLM's capabilities to iteratively refine prompts, increasing the likelihood of eliciting harmful responses to otherwise disallowed queries.

LLM-Virus: Evolutionary Jailbreak Attack on Large Language Models

Evaluated models: Claude 2, Claude 3.5 Haiku, GPT-3.5 Turbo +5 more

Diffusion-Driven LLM Jailbreak

Published Dec 23, 2024Analyzed Dec 28, 2024Source: arXiv

DiffusionAttacker exploits a vulnerability in Large Language Models (LLMs) allowing manipulation of prompts to elicit harmful responses, even when the model incorporates safety mechanisms. The attack leverages a sequence-to-sequence diffusion model to rewrite harmful prompts, making them appear harmless to the LLM's internal representation while preserving their original semantic meaning. This bypasses safety filters and elicits undesired outputs.

DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak

Evaluated models: Alpaca 7B, Claude 3.5 Sonnet, GPT-3.5 Turbo +4 more

Embodied AI Policy Jailbreak

Published Dec 21, 2024Analyzed Jan 26, 2025Source: arXiv

LLM-based planning modules in embodied AI systems are vulnerable to Policy Executable (POEX) jailbreak attacks. Attackers can inject carefully crafted adversarial suffixes into user instructions, causing the LLM to generate and execute harmful policies in both simulated and real-world environments. The attacks bypass safety mechanisms by using optimized, human-readable suffixes that evade perplexity-based detection.

POEX: Policy Executable Embodied AI Jailbreak Attacks

Evaluated models: Claude 3.5 Sonnet, GPT-4, GPT-4 Turbo +9 more

Multimodal LLM Jailbreak

Published Dec 21, 2024Analyzed Dec 29, 2024Source: arXiv

A hybrid multimodal jailbreaking attack, dubbed JMLLM, exploits vulnerabilities in 13 popular large language models (LLMs) across text, image, and speech modalities. The attack leverages alternating translation, word encryption, feature collapse in images, and harmful text injection to bypass safety mechanisms and elicit harmful responses. Success rates vary across LLMs and modalities, with some models exhibiting significantly higher vulnerability than others.

Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Models

Evaluated models: Claude 1, Claude 2, ERNIE 3.5 Turbo +10 more

Contextual Adversarial Prompts

Published Dec 20, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to human-readable adversarial prompts crafted using situational context derived from movie scripts. These prompts, which combine a malicious prompt, a seemingly innocuous adversarial insertion, and relevant contextual information, can bypass LLMs' safety mechanisms and elicit harmful responses. The technique leverages the LLM's ability to understand context and generate responses consistent with that context to mask the malicious intent. The…

Human-Readable Adversarial Prompts: An Investigation into LLM Vulnerabilities Using Situational Context

Evaluated models: Flan-t5 Large, Gemini 1.5 Pro, Gemma 2B IT +16 more

Preference-Optimized Jailbreak

Published Dec 20, 2024Analyzed Jan 26, 2025Source: arXiv

JailPO is a black-box attack framework that leverages preference optimization to generate effective jailbreak prompts for aligned LLMs. The attack automatically generates prompts, bypassing safety mechanisms and eliciting harmful or undesirable responses from the target LLM. The framework includes three attack patterns (QEPrompt, TemplatePrompt, MixAsking) with varying degrees of effectiveness and risk.

JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs

Evaluated models: GPT-3.5 Turbo

Linked-Task LLM Jailbreak

Published Dec 19, 2024Analyzed Dec 28, 2024Source: arXiv

A novel jailbreak paradigm, Simple Assistive Task Linkage (SATA), circumvents LLM safeguards by masking harmful keywords in a malicious query and using a secondary, simple assistive task (e.g., masked language modeling or element lookup by position) to convey the masked keywords' semantics to the LLM. This distracts the LLM and allows it to bypass safety checks, leading to the generation of harmful responses.

SATA: A Paradigm for LLM Jailbreak via Simple Assistive Task Linkage

Evaluated models: Claude-v2, GPT-3.5 Turbo, GPT-4o +3 more

One-Step Model Jailbreak

Published Dec 17, 2024Analyzed Dec 28, 2024Source: arXiv

A vulnerability in LLMs allows attackers to bypass safety mechanisms by crafting prompts that disguise malicious intent as a "defense" against harmful content. The attack, Reverse Embedded Defense Attack (REDA), leverages the model's own defensive capabilities to generate harmful outputs while masking the malicious intent within the response structure. This allows for successful jailbreaks in a single iteration, without requiring model-specific prompt engineering.

Jailbreaking? One Step Is Enough!

Evaluated models: GLM 4 9B Chat, GPT-3.5, Llama 2 13B +3 more

Related research categories