Attack Context

Black-box Research

Attacks requiring no knowledge of model internals

736 matching entries

Black-box research results

736 entries

LLM Contextual Divergence Jailbreak

Published 11/1/2024

Analyzed 12/28/2024

Large Language Models (LLMs) are vulnerable to a jailbreak attack that leverages the model's ability to generate diverse and obfuscated prompts to bypass safety constraints. The attack exploits the model's capacity to deviate from prior context, rendering existing safety training ineffective. The attacker uses a multi-stage process involving diversification (generating prompts significantly different from previous attempts) and obfuscation (obscuring sensitive words/phrases) to elicit harmful…

Diversity Helps Jailbreak Large Language Models

Evaluated models: Gemini 1.5 Pro, GPT-3.5 Turbo, GPT-4 +6 more

Source: arXiv

Language Game Jailbreaks

Published 11/1/2024

Analyzed 1/26/2025

Large Language Models (LLMs) are vulnerable to jailbreak attacks using language games, which manipulate input prompts through structured linguistic alterations (e.g., Ubbi Dubbi, custom letter insertion rules) to bypass safety mechanisms. These games obfuscate malicious intent while maintaining human readability, causing LLMs to generate unsafe content.

Playing Language Game with LLMs Leads to Jailbreaking

Evaluated models: Claude 3.5 Sonnet, GPT-4o, GPT-4o Mini +1 more

Source: arXiv

Multi-Round Jailbreak Agent

Published 11/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to multi-round jailbreak attacks which leverage a heuristic search process to progressively elicit harmful content. The attack decomposes a harmful query into multiple, seemingly innocuous sub-queries, iteratively refining the prompts based on the LLM's responses and employing psychological strategies to bypass safety mechanisms. This allows for the circumvention of single-round detection methods and elicitation of responses containing prohibited…

MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialogue

Evaluated models: DALL-E 3, GPT-3.5 Turbo, GPT-4 +4 more

Source: arXiv

Multi-Step Moralized Jailbreak

Published 11/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to multi-step "moralized" jailbreak prompts that bypass their safety guardrails. These prompts, while appearing ethical individually, cumulatively create a context that elicits verbally aggressive and harmful content generation. The attack leverages the LLMs' inability to fully understand the cumulative context and intent across multiple prompts.

" Moralized" Multi-Step Jailbreak Prompts: Black-Box Testing of Guardrails in Large Language Models for Verbal Attacks

Evaluated models: Claude 3.5 Sonnet, GPT-4o, Grok 2 +1 more

Source: arXiv

Nonlinear Prompt Jailbreak Features

Published 11/1/2024

Analyzed 12/28/2024

Large language models (LLMs) are vulnerable to jailbreak attacks exploiting nonlinear features within prompt encodings. These features, not detectable by linear methods, allow adversaries to reliably elicit harmful outputs despite safety training. Different attack methods leverage distinct nonlinear features, limiting the transferability of detection and mitigation techniques.

What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks

Evaluated models: Gemma 7B IT, Llama 3 8B Instruct

Source: arXiv

RL-Tuned LLM Jailbreak

Published 11/1/2024

Analyzed 12/29/2024

LLMStinger uses a reinforcement-learning loop to fine-tune an attacker model that generates adversarial suffixes for jailbreak prompts. Because the approach does not require white-box access to the target, it can adapt existing attacks for both open- and closed-source assistants and bypass otherwise effective refusal behavior.

LLMStinger: Jailbreaking LLMs using RL fine-tuned LLMs

Evaluated models: Claude 2, Gemma 2B IT, GPT-3.5 Turbo +3 more

Source: arXiv

SQL Injection Jailbreak

Published 11/1/2024

Analyzed 12/28/2024

A novel SQL Injection Jailbreak (SIJ) vulnerability allows attackers to bypass safety mechanisms in Large Language Models (LLMs) by manipulating the structure of input prompts. The attack leverages the model's processing of system prompts, user prefixes, user prompts, and assistant prefixes to effectively "comment out" the expected response prefix and inject harmful instructions, causing the LLM to generate unsafe content. This vulnerability exploits the external properties of the LLM…

SQL Injection Jailbreak: a structural disaster of large language models

Evaluated models: DeepSeek LLM 7B Chat, Llama 2 7B Chat, Llama 3.1 8B Instruct +2 more

Source: arXiv

Sequential Prompt Jailbreak

Published 11/1/2024

Analyzed 1/26/2025

Large Language Models (LLMs) are vulnerable to "SequentialBreak," a jailbreak attack where embedding a harmful prompt within a chain of benign prompts in a single query can bypass LLM safety features. The LLM's attention mechanism prioritizes the benign prompts, allowing the harmful prompt to be processed without triggering safety mitigations.

SequentialBreak: Large Language Models Can be Fooled by Embedding Jailbreak Prompts into Sequential Prompt Chains

Evaluated models: Not reported

Source: arXiv

Stochastic Monkey Jailbreak

Published 11/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) employing safety alignment mechanisms are vulnerable to a bypass attack using simple, stochastic random augmentations of input prompts. The attack leverages the inherent brittleness of safety alignment to minor, randomly introduced modifications in the input, causing the LLM to generate unsafe outputs despite its safety training. Character-level augmentations prove significantly more effective than string insertions.

Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignment

Evaluated models: GPT-4o, Llama 2 13B Chat, Llama 2 7B Chat +12 more

Source: arXiv

VLM RedTeaming Jailbreak

Published 11/1/2024

Analyzed 12/29/2024

Large Vision-Language Models (VLMs) are vulnerable to a novel black-box jailbreak attack, IDEATOR, which leverages a separate VLM to generate malicious image-text pairs. The attacker VLM iteratively refines its prompts based on the target VLM's responses, bypassing safety mechanisms by generating contextually relevant and visually subtle malicious prompts.

IDEATOR: Jailbreaking and Benchmarking Large Vision-Language Models Using Themselves

Evaluated models: MiniGPT-4 Vicuna 13B, InstructBLIP, Chameleon +10 more

Source: arXiv

Related research categories