Attack Context

Black-box Research

Attacks requiring no knowledge of model internals

736 matching entries

Black-box research results

736 entries

Humorous LLM Jailbreak

Published Apr 9, 2025Analyzed Apr 12, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to a jailbreaking attack leveraging humorous prompts. Embedding an unsafe request within a humorous context, using a fixed template, bypasses built-in safety mechanisms and elicits unsafe responses. The attack's success relies on a balance; too little or too much humor reduces effectiveness.

Bypassing Safety Guardrails in LLMs Using Humor

Evaluated models: Gemma 3 27B IT, Llama 3.1 8B Instruct, Llama 3.3 70B Instruct+1 more

Gemma 3 27B IT
Llama 3.1 8B Instruct
Llama 3.3 70B Instruct
Mixtral 8x7B Instruct

Benign-Prompt Jailbreak

Published Apr 8, 2025Analyzed Apr 12, 2025Source: arXiv

Large Language Models (LLMs) exhibit Defense Threshold Decay (DTD): generating substantial benign content shifts the model's attention from the input prompt to prior outputs, increasing susceptibility to jailbreak attacks. The "Sugar-Coated Poison" (SCP) attack exploits this by first generating benign content, then transitioning to malicious output.

Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking

Evaluated models: Claude 3.5 Sonnet, DeepSeek R1, GPT-3.5 Turbo+3 more

Claude 3.5 Sonnet
DeepSeek R1
GPT-3.5 Turbo
GPT-4
Llama 3.1 405B
Mixtral 8x22B

Generative Reward Hacking

Published Apr 8, 2025Analyzed Dec 9, 2025Source: arXiv

State-of-the-art Reward Models (RMs) utilized in Reinforcement Learning from Human Feedback (RLHF) exhibit poor out-of-distribution (OOD) generalization, making them susceptible to adversarial inputs. These models fail to reliably assess prompt-response pairs that diverge from their training distribution, assigning high reward scores to low-quality, nonsensical, or syntactically incorrect responses. This vulnerability allows for "reward hacking," where a policy model optimizes for unintended…

Adversarial training of reward models

Evaluated models: Llama 3.1 8B, Llama 3.3 70B, DeepSeek R1+1 more

Llama 3.1 8B
Llama 3.3 70B
DeepSeek R1
Gemma 2 27B

Single-Shot RAG Poisoning

Published Apr 4, 2025Analyzed Feb 22, 2026Source: arXiv

Retrieval-Augmented Generation (RAG) systems are vulnerable to a targeted corpus poisoning attack known as "CorruptRAG". This vulnerability allows an attacker to manipulate the response of an LLM to a specific target query by injecting a single malicious document into the RAG knowledge database. Unlike traditional poisoning attacks that require flooding the retrieval results (top-N) with malicious content to outnumber correct information, CorruptRAG succeeds with a single retrieved document.

Practical poisoning attacks against retrieval-augmented generation

Evaluated models: GPT-3.5, GPT-4, GPT-4o

Multimodal Contextual Jailbreak

Published Apr 2, 2025Analyzed Apr 12, 2025Source: arXiv

Multimodal Large Language Models (MLLMs) are vulnerable to a jailbreaking attack, dubbed PiCo, that leverages token-level typographic attacks on images embedded within code-style instructions. The attack bypasses multi-tiered defense mechanisms, including input filtering and runtime monitoring, by exploiting weaknesses in the visual modality's integration with programming contexts. Harmful intent is concealed within visually benign image fragments and code instructions, circumventing safety…

PiCo: Jailbreaking Multimodal Large Language Models via Pictorial Code Contextualization

Evaluated models: Gemini 1.0 Pro Vision, GPT-4 Turbo, GPT-4o+2 more

Gemini 1.0 Pro Vision
GPT-4 Turbo
GPT-4o
GPT-4V
LLaVA 1.5

Multi-Accent Audio Jailbreak

Published Apr 1, 2025Analyzed Apr 12, 2025Source: arXiv

Multilingual and multi-accent audio inputs, combined with acoustic adversarial perturbations (reverberation, echo, whisper effects), can bypass safety mechanisms in Large Audio Language Models (LALMs), causing them to generate unsafe or harmful outputs. The vulnerability is amplified by the interaction between acoustic and linguistic variations, particularly in languages with less training data.

Multilingual and Multi-Accent Jailbreaking of Audio LLMs

Evaluated models: DIVA Llama 3 v0 8B, MERaLion AudioLLM, MiniCPM-o 2.6+2 more

DIVA Llama 3 v0 8B
MERaLion AudioLLM
MiniCPM-o 2.6
Qwen 2 Audio
Ultravox

Multi-Agent Prompt Permutation Attack

Published Mar 31, 2025Analyzed Apr 12, 2025Source: arXiv

A vulnerability in multi-agent Large Language Model (LLM) systems allows for a permutation-invariant adversarial prompt attack. By strategically partitioning adversarial prompts and routing them through a network topology, an attacker can bypass distributed safety mechanisms, even those with token bandwidth limitations and asynchronous message delivery. The attack optimizes prompt propagation as a maximum-flow minimum-cost problem, maximizing success while minimizing detection.

Agents Under Siege: Breaking Pragmatic Multi-Agent LLM Systems with Optimized Prompt Attacks

Evaluated models: DeepSeek R1 Distill, Gemma 2 9B, Llama 2 7B+7 more

DeepSeek R1 Distill
Gemma 2 9B
Llama 2 7B
Llama 3.1 8B
Llama Guard
Llama Guard 2 8B
Llama Guard 3 1B
Llama Guard 3 8B
Mistral 7B
Prompt Guard

Recommender Memory Update Corruption

Published Mar 31, 2025Analyzed Dec 30, 2025Source: arXiv

Improper input validation in the memory module of Large Language Model (LLM)-powered agentic Recommender Systems (RS) allows remote attackers to perform indirect prompt injection via adversarial item descriptions. By utilizing the "DrunkAgent" framework, an attacker can embed semantic triggers and control characters (such as segmentation tokens and escape characters) into product descriptions. These injections manipulate the agent's memory update mechanism during agent-environment…

DrunkAgent: Stealthy Memory Corruption in LLM-Powered Recommender Agents

Evaluated models: GPT-4, o1, Llama 3 8B

Schema-Guided LLM Jailbreak

Published Mar 31, 2025Analyzed Apr 12, 2025Source: arXiv

Large Language Models (LLMs) with structured output APIs (e.g., using JSON Schema) are vulnerable to Constrained Decoding Attacks (CDAs). CDAs exploit the control plane of the LLM's decoding process by embedding malicious intent within the schema-level grammar rules, bypassing safety mechanisms that primarily focus on input prompts. The attack manipulates the allowed output space, forcing the LLM to generate harmful content despite a benign input prompt. One instance of a CDA is the Chain Enum…

Output Constraints as Attack Surface: Exploiting Structured Generation to Bypass LLM Safety Mechanisms

Evaluated models: Gemini 2.0 Flash, Gemma 2 9B, GPT-4o+5 more

Gemini 2.0 Flash
Gemma 2 9B
GPT-4o
GPT-4o Mini
Llama 3.1 8B
Mistral Nemo
Phi 3.5 MoE
Qwen 2.5 32B

Zero-Shot Embedding Leak

Published Mar 31, 2025Analyzed Dec 30, 2025Source: arXiv

A Universal Zero-shot Embedding Inversion vulnerability exists in vector databases and embedding-based retrieval systems. The flaw allows an attacker to reconstruct original plaintext documents from their vector embeddings without requiring access to the original training data or training an embedding-specific inversion model. The attack, identified as "ZSinvert," leverages a multi-stage adversarial decoding process: (1) a cosine-similarity guided beam search using a Large Language Model (LLM)…

Universal Zero-shot Embedding Inversion

Evaluated models: Qwen 2 5B

Related research categories