Attack Type

Jailbreak Research

Methods for bypassing model safety measures

704 matching entries

Jailbreak research results

704 entries

Adaptive Position Jailbreak

Published Sep 11, 2024Analyzed Dec 29, 2024Source: arXiv

AdaPPA is a jailbreak attack that exploits the varying levels of alignment protection in LLMs at different output positions. It leverages the model's instruction-following capabilities by pre-filling the output with carefully crafted "safe" content, creating a perceived completion and lowering the model's guard before generating malicious content. The attack's effectiveness relies on the adaptive generation of both safe and harmful pre-fill content, strategically placed to exploit weaknesses…

AdaPPA: Adaptive Position Pre-Fill Jailbreak Attack Approach Targeting LLMs

Evaluated models: ChatGLM3 6B, GPT-4o, GPT-4o Mini +5 more

Single-Turn LLM Jailbreak

Published Sep 4, 2024Analyzed Dec 28, 2024Source: arXiv

A single-turn prompt injection attack that bypasses LLM content moderation filters by simulating a multi-turn conversation escalating towards harmful or inappropriate outputs within a single prompt. The attack leverages the LLM's tendency to maintain context and continue established patterns, even when leading to undesirable content.

Well, that escalated quickly: The Single-Turn Crescendo Attack (STCA)

Evaluated models: GPT-4, GPT-4o, GPT-4o Mini +4 more

Attack LLMs with Toxic Answers

Published Aug 27, 2024Analyzed Aug 16, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to a targeted jailbreak attack, termed Atoxia, which can force the generation of specific harmful content. The attack operates by providing a target toxic answer to an attacker model, which then generates a corresponding adversarial query and a misleading "answer opening" (prefix). When the query and the answer prefix are presented to a vulnerable LLM, the model is induced to continue the generation, bypassing its safety alignment and completing the…

Atoxia: Red-teaming Large Language Models with Target Toxic Answers

Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4o +5 more

Random Token T2I Jailbreak

Published Aug 25, 2024Analyzed Dec 29, 2024Source: arXiv

A heuristic token search attack, termed HTS-Attack, can bypass safety mechanisms in text-to-image (T2I) models, allowing generation of NSFW content. The attack iteratively replaces tokens in a malicious prompt with semantically similar tokens from the model's vocabulary, avoiding detection by prompt and image checkers. The method leverages a surrogate CLIP model to maintain semantic similarity to the target NSFW prompt.

Rt-attack: Jailbreaking text-to-image models via random token

Evaluated models: Clip-vit-base-patch32, DALL-E 3, GPT 3.5-turbo-instruct +4 more

LLM Adversarial Suffix Optimization

Published Aug 21, 2024Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a novel black-box jailbreaking attack, ECLIPSE, which leverages the LLM's own capabilities as an optimizer to generate adversarial suffixes. ECLIPSE iteratively refines these suffixes based on a harmfulness score, bypassing the need for pre-defined affirmative phrases used in previous optimization-based attacks. This allows for effective jailbreaking even with limited interaction and without white-box access to the LLM's internal parameters.

Unlocking Adversarial Suffix Optimization Without Affirmative Phrases: Efficient Black-box Jailbreaking via LLM as Optimizer

Evaluated models: Falcon 7B Instruct, GPT-3.5 Turbo, Llama 2 7B Chat +1 more

Adversarial Unlearning Bypass

Published Aug 20, 2024Analyzed Jan 26, 2025Source: arXiv

Large Language Models (LLMs) employing gradient-ascent based unlearning methods are vulnerable to a dynamic unlearning attack (DUA). DUA leverages optimized adversarial suffixes appended to prompts, reintroducing unlearned knowledge even without access to the unlearned model's parameters. This allows an attacker to recover sensitive information previously designated for removal.

Towards robust knowledge unlearning: An adversarial framework for assessing and improving unlearning robustness in large language models

Evaluated models: Llama 2 7B Chat, Llama 3 8B Instruct, Llama 3.1 8B Instruct

Carrier Article Jailbreak

Published Aug 20, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a novel jailbreak attack that leverages "neural carrier articles." This attack injects a prohibited query into a benign article generated by a secondary LLM, designed to be semantically similar to the prohibited query but not trigger the primary LLM's safety mechanisms. The secondary LLM generates articles based on hypernyms derived from the prohibited query, thus subtly shifting attention weights within the primary LLM, bypassing its safeguards.

Hide Your Malicious Goal Into Benign Narratives: Jailbreak Large Language Models through Neural Carrier Articles

Evaluated models: GPT-3.5 Turbo, GPT-4, Llama 2 7B +1 more

LCCT Data Extraction & Jailbreak

Published Aug 20, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Model (LLM)-based Code Completion Tools (LCCTs), such as GitHub Copilot and Amazon Q, are vulnerable to jailbreaking and training data extraction attacks due to their unique workflows and reliance on proprietary code datasets. Jailbreaking attacks exploit the LLM's ability to generate harmful content by embedding malicious prompts within various code components (filenames, comments, variable names, function calls). Training data extraction attacks leverage the LLM's tendency to…

Security Attacks on LLM-based Code Completion Tools

Evaluated models: GPT 3.5-turbo-0125, GPT-4 Turbo-2024-04-09, GPT-4o-2024-05-13

Perceptual Text-to-Image Jailbreak

Published Aug 20, 2024Analyzed Dec 29, 2024Source: arXiv

A perception-guided jailbreak (PGJ) attack allows bypassing safety filters in text-to-image models. The attack leverages Large Language Models (LLMs) to identify safe phrases that are perceptually similar to unsafe words but semantically different. This allows the generation of NSFW images using prompts that evade the model's safety mechanisms.

Perception-guided jailbreak against text-to-image models

Evaluated models: Cogview3, Dall-e 2, DALL-E 3 +5 more

Synthetic LLM Jailbreak Dataset

Published Aug 14, 2024Analyzed Jul 14, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to jailbreaking attacks leveraging synthetically generated prompts. A novel pipeline, SAGE-RT, generates a diverse dataset of 51,000 prompt-response pairs designed to exploit LLMs' vulnerabilities across various categories of harmfulness. These prompts successfully jailbreak state-of-the-art LLMs in a significant percentage of tested sub-categories, including 100% of macro-categories for certain models like GPT-4 and GPT-3.5-turbo. The vulnerability…

Sage-rt: Synthetic alignment data generation for safety evaluation and red teaming

Evaluated models: Claude 3.5 Sonnet, Gemma 7B IT, GPT-3.5 Turbo +8 more

Related research categories