Attack Type

Jailbreak Vulnerabilities

Methods for bypassing model safety measures

703 matching entries

Jailbreak research results

703 entries

Symbolic Math Jailbreak

Updated 12/29/2024

Large Language Models (LLMs) are vulnerable to a jailbreaking attack, termed "MathPrompt," which leverages the models' ability to process symbolic mathematics to bypass built-in safety mechanisms. The attack encodes harmful natural language prompts into mathematically formulated problems, causing the LLM to generate unsafe outputs while ostensibly solving a mathematical problem.

Jailbreaking Large Language Models with Symbolic Mathematics

Affects: Claude 3 Haiku, Claude 3 Opus, Claude 3 Sonnet +10 more

Source: arXiv

Adversarial Unlearning Bypass

Updated 1/26/2025

Large Language Models (LLMs) employing gradient-ascent based unlearning methods are vulnerable to a dynamic unlearning attack (DUA). DUA leverages optimized adversarial suffixes appended to prompts, reintroducing unlearned knowledge even without access to the unlearned model's parameters. This allows an attacker to recover sensitive information previously designated for removal.

Towards robust knowledge unlearning: An adversarial framework for assessing and improving unlearning robustness in large language models

Affects: Llama 2 7B Chat, Llama 3 8B Instruct, Llama 3.1 8B Instruct

Source: arXiv

Attack LLMs with Toxic Answers

Updated 8/16/2025

Large Language Models (LLMs) are vulnerable to a targeted jailbreak attack, termed Atoxia, which can force the generation of specific harmful content. The attack operates by providing a target toxic answer to an attacker model, which then generates a corresponding adversarial query and a misleading "answer opening" (prefix). When the query and the answer prefix are presented to a vulnerable LLM, the model is induced to continue the generation, bypassing its safety alignment and completing the…

Atoxia: Red-teaming Large Language Models with Target Toxic Answers

Affects: GPT-3.5 Turbo, GPT-4, GPT-4o +5 more

Source: arXiv

Carrier Article Jailbreak

Updated 12/29/2024

Large Language Models (LLMs) are vulnerable to a novel jailbreak attack that leverages "neural carrier articles." This attack injects a prohibited query into a benign article generated by a secondary LLM, designed to be semantically similar to the prohibited query but not trigger the primary LLM's safety mechanisms. The secondary LLM generates articles based on hypernyms derived from the prohibited query, thus subtly shifting attention weights within the primary LLM, bypassing its safeguards.

Hide Your Malicious Goal Into Benign Narratives: Jailbreak Large Language Models through Neural Carrier Articles

Affects: GPT-3.5 Turbo, GPT-4, Llama 2 7B +1 more

Source: arXiv

Composable Jailbreak Synthesis

Updated 12/28/2024

Large Language Models (LLMs) are vulnerable to composable jailbreak attacks, allowing bypass of safety filters through the chaining of multiple prompt transformations. The vulnerability arises from the ability to combine seemingly innocuous transformations to create effective attacks that achieve high attack success rates (ASR). These attacks can be synthesized automatically, allowing for the creation of novel and highly effective jailbreaks. Specifically, using the h4rm3l framework, attacks…

h4rm3l: A dynamic benchmark of composable jailbreak attacks for llm safety assessment

Affects: Claude 3 Haiku, Claude 3 Sonnet, GPT-3.5 Turbo +3 more

Source: arXiv

Contextual Fusion Jailbreak

Updated 12/29/2024

Large Language Models (LLMs) are vulnerable to a multi-turn context-based jailbreak attack, termed Context Fusion Attack (CFA). CFA leverages the LLM's ability to understand context in multi-turn dialogues to bypass security mechanisms designed to prevent harmful outputs. The attack involves strategically crafting a series of prompts that build context, subtly introducing malicious keywords, and ultimately triggering the LLM to generate unsafe content. The malicious intent is masked within the…

Multi-Turn Context Jailbreak Attack on Large Language Models From First Principles

Affects: ChatGLM4, GPT-3.5 Turbo, GPT-4

Source: arXiv

Ensemble Jailbreak Technique

Updated 12/29/2024

The Ensemble Jailbreak (EnJa) attack exploits vulnerabilities in the safety mechanisms of large language models (LLMs) by combining prompt-level and token-level attacks. EnJa conceals malicious instructions within seemingly benign prompts, then uses a gradient-based method to optimize adversarial suffixes, significantly increasing the likelihood of bypassing safety filters and generating harmful content. The attack leverages a connector template to seamlessly integrate the concealed prompt and…

EnJa: Ensemble Jailbreak on Large Language Models

Affects: GPT-3.5 Turbo, GPT-4, Llama 2 13B +3 more

Source: arXiv

GCG Suffix Data Exfiltration

Updated 3/24/2025

A Cross-Prompt Injection Attack (XPIA) can be amplified by appending a Greedy Coordinate Gradient (GCG) suffix to the malicious injection. This increases the likelihood that a Large Language Model (LLM) will execute the injected instruction, even in the presence of a user's primary instruction, leading to data exfiltration. The success rate of the attack depends on the LLM's complexity; medium-complexity models show increased vulnerability.

WHITE PAPER: A Brief Exploration of Data Exfiltration using GCG Suffixes

Affects: GPT-3.5 Turbo, GPT-4o, Phi 3 Mini

Source: arXiv

Kov: MDP-Based LLM Jailbreak

Updated 12/29/2024

Large Language Models (LLMs) are vulnerable to naturalistic adversarial attacks crafted using Markov Decision Processes (MDPs) and Monte Carlo Tree Search (MCTS). These attacks generate natural-language prompts that elicit harmful, violent, or discriminatory responses from the LLMs, even those with built-in safety mechanisms. The attacks are transferable across different LLMs, demonstrating a generalized vulnerability.

Kov: Transferable and Naturalistic Black-Box LLM Attacks using Markov Decision Processes and Tree Search

Affects: FastChat-T5 3B, GPT-3.5 Turbo, GPT-4 +1 more

Source: arXiv

LCCT Data Extraction & Jailbreak

Updated 12/29/2024

Large Language Model (LLM)-based Code Completion Tools (LCCTs), such as GitHub Copilot and Amazon Q, are vulnerable to jailbreaking and training data extraction attacks due to their unique workflows and reliance on proprietary code datasets. Jailbreaking attacks exploit the LLM's ability to generate harmful content by embedding malicious prompts within various code components (filenames, comments, variable names, function calls). Training data extraction attacks leverage the LLM's tendency to…

Security Attacks on LLM-based Code Completion Tools

Affects: GPT 3.5-turbo-0125, GPT-4 Turbo-2024-04-09, GPT-4o-2024-05-13

Source: arXiv

Related research categories