Attack Type

Jailbreak Vulnerabilities

Methods for bypassing model safety measures

703 matching entries

Jailbreak research results

703 entries

Stochastic Monkey Jailbreak

Updated 12/29/2024

Large Language Models (LLMs) employing safety alignment mechanisms are vulnerable to a bypass attack using simple, stochastic random augmentations of input prompts. The attack leverages the inherent brittleness of safety alignment to minor, randomly introduced modifications in the input, causing the LLM to generate unsafe outputs despite its safety training. Character-level augmentations prove significantly more effective than string insertions.

Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignment

Affects: GPT-4o, Llama 2 13B Chat, Llama 2 7B Chat +12 more

Source: arXiv

Universal LLM Output Control

Updated 12/29/2024

Large Language Models (LLMs) are vulnerable to a novel prompt injection attack using universal and context-independent triggers. These triggers, once discovered for a specific LLM, allow precise control over the model's output regardless of the prompt context or desired output content, enabling adversaries to force the generation of arbitrary text. The attack utilizes a gradient-based optimization technique to discover these triggers.

Universal and Context-Independent Triggers for Precise Control of LLM Outputs

Affects: Llama 3 70B Instruct, Llama 3 8B Instruct, Llama 3.1 70B Instruct +7 more

Source: arXiv

VLM RedTeaming Jailbreak

Updated 12/29/2024

Large Vision-Language Models (VLMs) are vulnerable to a novel black-box jailbreak attack, IDEATOR, which leverages a separate VLM to generate malicious image-text pairs. The attacker VLM iteratively refines its prompts based on the target VLM's responses, bypassing safety mechanisms by generating contextually relevant and visually subtle malicious prompts.

IDEATOR: Jailbreaking and Benchmarking Large Vision-Language Models Using Themselves

Affects: MiniGPT-4 Vicuna 13B, InstructBLIP, Chameleon +10 more

Source: arXiv

Visual Jailbreak via Multi-Loss

Updated 12/29/2024

Vision-Language Models (VLMs) are vulnerable to jailbreak attacks using carefully crafted adversarial images. Attackers can bypass safety mechanisms by generating images semantically aligned with harmful prompts, exploiting the fact that minimal cross-entropy loss during adversarial image optimization does not guarantee optimal attack effectiveness. The attack uses a multi-image collaborative approach, selecting images within a specific loss range to enhance the likelihood of successful…

Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models

Affects: LLaVA 2, MiniGPT-4

Source: arXiv

Zeroth-Order MLLM Jailbreak

Updated 12/29/2024

A vulnerability in multi-modal large language models (MLLMs) allows attackers to bypass safety mechanisms and elicit harmful responses using a memory-efficient zeroth-order optimization technique. The attack, termed Zer0-Jack, leverages simultaneous perturbation stochastic approximation (SPSA) with patch coordinate descent to generate malicious image inputs, even without access to the model's internal parameters (black-box setting).

Zer0-Jack: A Memory-efficient Gradient-based Jailbreaking Method for Black-box Multi-modal Large Language Models

Affects: GPT-4o, Inf-mllm1, LLaVA 1.5 +1 more

Source: arXiv

Attention-Based LLM Jailbreak

Updated 12/29/2024

Large Language Models (LLMs) are vulnerable to attention-based jailbreak attacks. Attackers can craft prompts that strategically divert the LLM's attention away from sensitive words, causing the model to overlook malicious intent and generate harmful content. This occurs by leveraging the LLM's attention mechanism to focus on benign parts of the prompt while embedding harmful queries within a seemingly harmless context. The success of the attack is correlated with specific attention…

Feint and Attack: Attention-Based Strategies for Jailbreaking and Protecting LLMs

Affects: Claude 3 Haiku, GPT-4, Llama 2 13B Chat +2 more

Source: arXiv

Attention-Guided Jailbreak

Updated 12/29/2024

Large Language Models (LLMs) are vulnerable to jailbreaking attacks that manipulate attention scores to redirect the model's focus away from safety protocols. The AttnGCG attack method increases the attention score on adversarial suffixes within the input prompt, causing the model to prioritize the malicious content over safety guidelines, leading to the generation of harmful outputs.

AttnGCG: Enhancing jailbreaking attacks on LLMs with attention manipulation

Affects: Gemini 1.5 Flash, Gemini Pro, Gemini 1.5 Pro Latest +6 more

Source: arXiv

Autonomous Jailbreak Agent

Updated 12/28/2024

Large Language Models (LLMs) are vulnerable to jailbreak attacks using autonomously discovered strategies. AutoDAN-Turbo, a black-box attack method, demonstrates the ability to discover novel and highly effective jailbreak strategies without human intervention, achieving a high success rate (e.g., 88.5% on GPT-4-1106-turbo) in eliciting harmful or unsafe responses from LLMs. The attack leverages a lifelong learning agent to iteratively refine attack strategies based on model responses…

Autodan-turbo: A lifelong agent for strategy self-exploration to jailbreak llms

Affects: Gemini Pro, Gemma 7B IT, GPT-4-1106-turbo +5 more

Source: arXiv

Benign Mirroring Jailbreak

Updated 12/29/2024

Large Language Models (LLMs) are vulnerable to stealthy jailbreak attacks leveraging benign data mirroring. Attackers train a local "mirror model" on benign data obtained from the target LLM. This mirror model, mimicking the target's behavior, is then used to generate adversarial prompts, which are subsequently deployed against the target LLM, bypassing content moderation systems due to the lack of overtly malicious content in the initial data gathering phase.

Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring

Affects: GPT-3.5 Turbo, GPT-4o Mini, Llama 2 Chat +1 more

Source: arXiv

Bijection-Based LLM Jailbreak

Updated 12/29/2024

Large Language Models (LLMs) are vulnerable to a novel "bijection learning" attack that leverages in-context learning to teach the model a custom string-to-string encoding, bypassing built-in safety mechanisms. The attack encodes harmful queries, sends them to the model, and decodes the response, effectively circumventing safety filters. The complexity of the encoding can be controlled, adapting the attack to various LLMs; more capable models are more susceptible to complex encodings.

Endless Jailbreaks with Bijection Learning

Affects: Claude 3 Haiku, Claude 3 Opus, Claude 3.5 Sonnet +5 more

Source: arXiv

Related research categories