Attack Context

Black-box Research

Attacks requiring no knowledge of model internals

736 matching entries

Black-box research results

736 entries

LLM Quality-Diversity Red-Teaming

Published Jun 8, 2025Analyzed Dec 9, 2025Source: arXiv

Large Language Models (LLMs), including Llama-3, Gemma-2, and Qwen2.5, are vulnerable to automated adversarial attacks generated via a Quality-Diversity Red-Teaming (QDRT) framework. This vulnerability arises from the models' inability to robustly defend against attackers trained via behavior-conditioned reinforcement learning that optimize for specific "goal-driven" behaviors. Unlike standard attacks that optimize solely for toxicity, QDRT trains a population of specialized attacker models to…

Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models

Evaluated models: Llama 3.2 3B Instruct, Llama 3.1 8B Instruct, Gemma 2 2B IT+5 more

Llama 3.2 3B Instruct
Llama 3.1 8B Instruct
Gemma 2 2B IT
Gemma 2 9B IT
Qwen 2.5 7B Instruct
Gemma 2 27B IT
Qwen 2.5 32B Instruct
Llama 3.3 70B Instruct

Bitstream Camouflage Jailbreak

Published Jun 3, 2025Analyzed Jul 14, 2025Source: arXiv

A novel black-box attack, dubbed BitBypass, exploits the vulnerability of aligned LLMs by camouflaging harmful prompts using hyphen-separated bitstreams. This bypasses safety alignment mechanisms by transforming sensitive words into their bitstream representations and replacing them with placeholders, in conjunction with a specially crafted system prompt that instructs the LLM to convert the bitstream back to text and respond as if given the original harmful prompt.

BitBypass: A New Direction in Jailbreaking Aligned Large Language Models with Bitstream Camouflage

Evaluated models: Claude 3.5 Sonnet, Gemini 1.5 Pro, GPT-4o+2 more

Claude 3.5 Sonnet
Gemini 1.5 Pro
GPT-4o
Llama 3.1 70B
Mixtral 8x22B

Agentic Red-Teaming Uncovers Novel Jailbreaks

Published Jun 1, 2025Analyzed Jul 28, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to jailbreaking through an agentic attack framework called Composition of Principles (CoP). This technique uses an attacker LLM (Red-Teaming Agent) to dynamically select and combine multiple human-defined, high-level transformations ("principles") into a single, sophisticated prompt. The composition of several simple principles, such as expanding context, rephrasing, and inserting specific phrases, creates complex adversarial prompts that can bypass…

CoP: Agentic Red-teaming for Large Language Models using Composition of Principles

Evaluated models: Claude 3.5 Sonnet, Gemini 1.5 Pro, Gemma 7B IT+11 more

Claude 3.5 Sonnet
Gemini 1.5 Pro
Gemma 7B IT
GPT-4
GPT-4 Turbo
GPT-4o
Grok 2
Llama 2 13B
Llama 2 13B Chat
Llama 2 70B Chat
Llama 2 7B Chat
Llama 3 70B Instruct
Llama 3 8B Instruct
o1

Phantom Token User Deception

Published May 30, 2025Analyzed Jan 14, 2026Source: arXiv

Large Language Models (LLMs) that utilize byte-stream parsing or structural extraction to process PDF files—specifically the OpenAI GPT and Anthropic Claude families—are vulnerable to adversarial text injection via imperceptible "phantom tokens." This vulnerability exploits the disconnect between how PDF viewers render documents for humans (visual layer) and how LLMs extract text from the PDF operator stream (data layer). Attackers can manipulate standard PDF text-showing operators (TJ and Tj)…

TRAPDOC: Deceiving LLM Users by Injecting Imperceptible Phantom Tokens into Documents

Evaluated models: GPT-4, o4-mini

Adaptive LLM Jailbreaking Strategy

Published May 29, 2025Analyzed Jul 14, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to adaptive jailbreaking attacks that exploit their semantic comprehension capabilities. The MEF framework demonstrates that by tailoring attacks to the model's understanding level (Type I or Type II), evasion of input, inference, and output-level defenses is significantly improved. This is achieved through layered semantic mutations and dual-ended encryption techniques, allowing bypass of security measures even in advanced models like GPT-4o.

Adaptive Jailbreaking Strategies Based on the Semantic Understanding Capabilities of Large Language Models

Evaluated models: GPT-4o, Llama 2 13B, Llama 2 7B

DNA Model Pathogen Synthesis

Published May 28, 2025Analyzed Jun 11, 2025Source: arXiv

DNA language models, such as the Evo series, are vulnerable to jailbreak attacks that coerce the generation of DNA sequences with high homology to known human pathogens. The GeneBreaker framework demonstrates this by using a combination of carefully crafted prompts leveraging high-homology non-pathogenic sequences and a beam search guided by pathogenicity prediction models (e.g., PathoLM) and log-probability heuristics. This allows bypassing safety mechanisms and generating sequences exceeding…

GeneBreaker: Jailbreak Attacks against DNA Language Models with Pathogenicity Guidance

Evaluated models: Evo1 7B, Evo2 1B, Evo2 7B+2 more

Evo1 7B
Evo2 1B
Evo2 7B
Evo2 40B
PathoLM

Hybrid Agent Prompt Injection

Published May 28, 2025Analyzed Dec 9, 2025Source: arXiv

Computer-Use Agents (CUAs) powered by Large Language Models (LLMs) operating in hybrid Web-OS environments are vulnerable to indirect prompt injection. Attackers can embed malicious natural language or code instructions within legitimate web content (e.g., social media forums, chat applications, shared cloud documents) that the agent processes during benign task execution. Due to the agent's inability to distinguish between trusted user instructions and untrusted environmental data, the CUA…

RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environments

Evaluated models: Claude 3.5 Sonnet, Claude 3.7 Sonnet, GPT-4o

Expanded Strategy Jailbreak

Published May 27, 2025Analyzed May 31, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to jailbreak attacks that exploit the model's inherent persuasive nature. A novel attack framework, CL-GSO, decomposes jailbreak strategies into four components (Role, Content Support, Context, Communication Skills), creating a significantly expanded strategy space compared to prior methods. This expanded space allows for the generation of prompts that bypass safety protocols with a success rate exceeding 90% on models previously considered…

Breaking the Ceiling: Exploring the Potential of Jailbreak Attacks through Expanding Strategy Space

Evaluated models: Claude 3.5 Sonnet, GPT-3.5 Turbo, GPT-4o+2 more

Claude 3.5 Sonnet
GPT-3.5 Turbo
GPT-4o
Llama 3 8B
Qwen 2.5 7B

LLM System Prompt Extraction

Published May 27, 2025Analyzed Dec 30, 2025Source: arXiv

Large Language Models (LLMs), including Llama-3, Falcon-3, Gemma-2, and GPT-4 variants, are susceptible to system prompt extraction attacks. The vulnerability exists due to the models' instruction-following nature, which allows remote attackers to bypass safety guardrails and retrieve the model's hidden system configuration (system prompt) verbatim. This is successfully exploited using an "Extended Sandwich Attack," where an adversarial extraction command is embedded between benign questions…

System Prompt Extraction Attacks and Defenses in Large Language Models

Evaluated models: GPT-4, GPT-4o, Llama 3 8B+2 more

GPT-4
GPT-4o
Llama 3 8B
Falcon 7B
Gemma 2 9B

Distilled Jailbreak Attacks

Published May 26, 2025Analyzed Jul 14, 2025Source: arXiv

A vulnerability in Large Language Models (LLMs) allows adversarial prompt distillation from a large language model (LLM) to a smaller language model (SLM), enabling efficient and stealthy jailbreak attacks. The attack leverages knowledge distillation techniques, reinforcement learning, and dynamic temperature control to transfer the LLM's ability to bypass safety mechanisms to a smaller, more easily deployable SLM. This allows for lower computational cost attacks with a potentially high…

Efficient and Stealthy Jailbreak Attacks via Adversarial Prompt Distillation from LLMs to SLMs

Evaluated models: BERT Base, Gemma 2 27B, Gemma 2 2B+8 more

BERT Base
Gemma 2 27B
Gemma 2 2B
GPT-3.5 Turbo
GPT-4
GPT-4o
Llama 2 13B
Llama 2 7B
Llama 3.2 1B
Vicuna 13B
Vicuna 7B

Related research categories