Attack Context

Black-box Research

Attacks requiring no knowledge of model internals

736 matching entries

Black-box research results

736 entries

Alignment-Based LLM Jailbreak

Published Dec 6, 2024Analyzed Jan 26, 2025Source: arXiv

Large Language Models (LLMs) employing reinforcement learning from human feedback (RLHF) for safety alignment are vulnerable to a novel "alignment-based" jailbreak attack. This attack leverages a best-of-N sampling approach with an adversarial LLM to efficiently generate prompts that bypass safety mechanisms and elicit unsafe responses from the target LLM, without requiring additional training or access to the target LLM's internal parameters. The attack exploits the inherent tension between…

LIAR: Leveraging Alignment (Best-of-N) to Jailbreak LLMs in Seconds

Evaluated models: Falcon 7B, GPT-2, Llama 3.1 8B+6 more

Falcon 7B
GPT-2
Llama 3.1 8B
Megatron 345M
Mistral 7B
Pythia 12B
Tiny Llama 1.1B
Vicuna 13B
Vicuna 7B

Best-of-N Prompt Augmentation

Published Dec 4, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) across multiple modalities (text, vision, audio) are vulnerable to a "Best-of-N" (BoN) jailbreaking attack. This attack repeatedly submits slightly modified versions of a harmful prompt (e.g., text with altered capitalization, images with modified text style, audio with altered pitch or speed) until a safety mechanism is bypassed and a harmful response is elicited. The effectiveness of the attack scales with the number of attempts (N). While individual…

Best-of-N Jailbreaking

Evaluated models: Claude 3 Opus, Claude 3.5 Sonnet, Cygnet+10 more

Claude 3 Opus
Claude 3.5 Sonnet
Cygnet
DiVA
Gemini 1.5 Pro
Gemini-1.5-flash-001
Gemini-1.5-pro-001
GPT-4o
GPT-4o Mini
GPT-4o Realtime
Llama 3 8B
Llama 3 8B Instruct
Llama 3.1 8B

Natural Prompt Jailbreaks

Published Dec 4, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) trained with safety fine-tuning are vulnerable to a novel attack, Response-Guided Question Augmentation (ReG-QA). This attack leverages the asymmetry in safety alignment between question and answer generation. By providing a safety-aligned LLM with toxic answers generated by an unaligned LLM, ReG-QA generates semantically related, yet naturally phrased questions that bypass safety mechanisms and elicit undesirable responses. The attack does not require adversarial…

Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?

Evaluated models: Gemma 2 27B IT, Gemma 2 9B IT, GPT-3.5 Turbo+6 more

Gemma 2 27B IT
Gemma 2 9B IT
GPT-3.5 Turbo
GPT-4 Turbo
GPT-4o
Mistral 7B Instruct v0.2
Mixtral-8x22B-instruct-v0.1
Palm-2-otter
Qwen 2.5 72B Instruct

Multi-Modal VLM Jailbreak

Published Nov 30, 2024Analyzed Jan 26, 2025Source: arXiv

A novel jailbreak attack, Multi-Modal Linkage (MML), exploits the vulnerability in Large Vision-Language Models (VLMs) by leveraging an "encryption-decryption" scheme across text and image modalities. MML encrypts malicious queries within images (e.g., using word replacement, image transformations) to bypass initial safety mechanisms. A subsequent text prompt guides the VLM to "decrypt" the content, eliciting harmful outputs. "Evil alignment," framing the attack within a video game scenario…

Jailbreak Large Visual Language Models Through Multi-Modal Linkage

Evaluated models: Claude 3.5 Sonnet, GPT-4o, GPT-4o Mini+1 more

Claude 3.5 Sonnet
GPT-4o
GPT-4o Mini
Qwen VL Max

Visual Jailbreak via Multi-Loss

Published Nov 27, 2024Analyzed Dec 29, 2024Source: arXiv

Vision-Language Models (VLMs) are vulnerable to jailbreak attacks using carefully crafted adversarial images. Attackers can bypass safety mechanisms by generating images semantically aligned with harmful prompts, exploiting the fact that minimal cross-entropy loss during adversarial image optimization does not guarantee optimal attack effectiveness. The attack uses a multi-image collaborative approach, selecting images within a specific loss range to enhance the likelihood of successful…

Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models

Evaluated models: LLaVA 2, MiniGPT-4

Multi-Step Moralized Jailbreak

Published Nov 23, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to multi-step "moralized" jailbreak prompts that bypass their safety guardrails. These prompts, while appearing ethical individually, cumulatively create a context that elicits verbally aggressive and harmful content generation. The attack leverages the LLMs' inability to fully understand the cumulative context and intent across multiple prompts.

" Moralized" Multi-Step Jailbreak Prompts: Black-Box Testing of Guardrails in Large Language Models for Verbal Attacks

Evaluated models: Claude 3.5 Sonnet, GPT-4o, Grok 2+1 more

Claude 3.5 Sonnet
GPT-4o
Grok 2
Llama 3.1 405B

Adversarial Suffix Jailbreak

Published Nov 21, 2024Analyzed Dec 29, 2024Source: arXiv

Research ID 5cdb4e1d

Large language models (LLMs) are vulnerable to adversarial suffix injection attacks. Maliciously crafted suffixes appended to otherwise benign prompts can cause the LLM to generate harmful or undesired outputs, bypassing built-in safety mechanisms. The attack leverages the model's sensitivity to input perturbations to elicit responses outside its intended safety boundaries.

GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMs

Evaluated models: Falcon 7B Instruct, GPT-3.5 Turbo, GPT-4o+5 more

Falcon 7B Instruct
GPT-3.5 Turbo
GPT-4o
GPT-4o Mini
Llama 2 7B Chat
Llama 3 8B Instruct
Llama 3.1 8B Instruct
Mistral 7B Instruct v0.3

Authority Citation Jailbreak

Published Nov 18, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) exhibit a bias towards authoritative sources, allowing attackers to bypass safety mechanisms by crafting prompts that include fabricated citations mimicking credible sources (e.g., research papers, GitHub repositories). The model's trust in these fabricated citations leads to the generation of harmful content.

The Dark Side of Trust: Authority Citation-Driven Jailbreak Attacks on Large Language Models

Evaluated models: Baichuan-13B, Claude-3(v3-haiku), GPT-3.5 Turbo+4 more

Baichuan-13B
Claude-3(v3-haiku)
GPT-3.5 Turbo
GPT-4 0613
GPT-4o
Llama 2 7B Chat
Llama 3 8B Instruct

Image-Based Safety Snowballing

Published Nov 18, 2024Analyzed Dec 29, 2024Source: arXiv

A vulnerability exists in several Large Vision-Language Models (LVLMs) where seemingly safe images, when combined with additional safe images and prompts using a specific attack methodology (Safety Snowball Agent), can trigger the generation of unsafe and harmful content. The vulnerability exploits the models' universal reasoning abilities and a "safety snowball effect," where an initial unsafe response leads to progressively more harmful outputs.

Safe+ Safe= Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Models

Evaluated models: GPT-4o, InternVL 2 40B, Qwen VL 2 72B+1 more

GPT-4o
InternVL 2 40B
Qwen VL 2 72B
VILA 1.5 40B

Language Game Jailbreaks

Published Nov 16, 2024Analyzed Jan 26, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to jailbreak attacks using language games, which manipulate input prompts through structured linguistic alterations (e.g., Ubbi Dubbi, custom letter insertion rules) to bypass safety mechanisms. These games obfuscate malicious intent while maintaining human readability, causing LLMs to generate unsafe content.

Playing Language Game with LLMs Leads to Jailbreaking

Evaluated models: Claude 3.5 Sonnet, GPT-4o, GPT-4o Mini+1 more

Claude 3.5 Sonnet
GPT-4o
GPT-4o Mini
Llama 3.1 70B

Related research categories