Attack Surface

Prompt layer Research

Security research involving prompt handling and processing

610 matching entries

Prompt layer research results

610 entries

LLM Guardrail Bypass

Published Jul 8, 2025Analyzed Dec 8, 2025Source: arXiv

Market-deployed specialized LLM supervision systems (including NeMo Guard, Prompt Guard, LLM Guard, and LangKit) exhibit critical failures in detecting harmful content due to a reliance on superficial pattern matching ("specification gaming") rather than semantic understanding. These systems fail to generalize to inputs that do not match specific training patterns, resulting in near-zero detection rates for straightforward harmful prompts in categories such as CBRN (Chemical, Biological…

The bitter lesson of misuse detection

Evaluated models: GPT-3.5, GPT-4, Claude 3.5 Sonnet+4 more

GPT-3.5
GPT-4
Claude 3.5 Sonnet
Grok 2
Gemini 1.5 Pro
Mistral Large
DeepSeek V3

Forged Assistant Message Jailbreak

Published Jul 7, 2025Analyzed Jan 14, 2026Source: arXiv

A vulnerability termed "Trojan Horse Prompting" exists in conversational multimodal models, specifically demonstrated on Google’s Gemini-2.0-flash-preview-image-generation. The vulnerability allows an attacker to bypass safety alignment mechanisms (RLHF and SFT) by manipulating the structural protocol of the conversational API. Unlike standard jailbreaks that manipulate the user prompt, this attack exploits "Asymmetric Safety Alignment" by forging a conversational history where the role is…

Trojan Horse Prompting: Jailbreaking Conversational Multimodal Models by Forging Assistant Message

Evaluated models: Gemini 2.0 Flash Preview Image Generation

Multi-Agent Mole Attack

Published Jul 7, 2025Analyzed Dec 9, 2025Source: arXiv

A vulnerability exists in Large Language Model (LLM)-based Multi-Agent Systems (MAS) that allows a malicious agent to covertly disrupt collaborative decision-making processes without triggering standard safety filters or anomaly detection. This "intention-hiding" attack occurs when an agent adopts a persona that appears linguistically fluent and role-consistent but strategically steers the group toward incorrect outcomes or resource exhaustion. The attacker leverages specific semantic…

Who's the Mole? Modeling and Detecting Intention-Hiding Malicious Agents in LLM-Based Multi-Agent Systems

Evaluated models: GPT-4o

Response-Primed LLM Jailbreak

Published Jul 7, 2025Analyzed Aug 31, 2025Source: arXiv

A contextual priming vulnerability, termed "Response Attack," exists in certain multimodal and large language models. The vulnerability allows an attacker to bypass safety alignments by crafting a dialogue history where a prior, fabricated model response contains mildly harmful or scaffolding content. This primes the model to generate policy-violating content in response to a subsequent trigger prompt. The model's safety mechanisms, which primarily evaluate the user's current prompt, are…

Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Models

Evaluated models: DeepSeek R1 Distill Llama 70B, Gemini 2.0 Flash, Gemini 2.5 Flash+5 more

DeepSeek R1 Distill Llama 70B
Gemini 2.0 Flash
Gemini 2.5 Flash
GPT-4.1
GPT-4o
Llama 3 70B Instruct
Llama 3 8B Instruct
QwQ 32B

Compositional Malware Generation

Published Jul 2, 2025Analyzed Dec 9, 2025Source: arXiv

Aligned Large Language Models (LLMs) exhibit a "compositional blindness" vulnerability wherein safety alignment mechanisms evaluate user prompts in isolation, failing to detect malicious intent when it is systematically decomposed into multiple benign-appearing sub-tasks. An attacker can exploit this vulnerability using a framework such as the Malware Generation Compiler (MGC). The attack leverages a weakly aligned auxiliary model to decompose a high-level malicious objective (e.g…

MGC: A Compiler Framework Exploiting Compositional Blindness in Aligned LLMs for Malware Generation

Evaluated models: Mistral 7B Instruct v0.3, GPT-4o Mini, Claude 3.5 Sonnet+1 more

Mistral 7B Instruct v0.3
GPT-4o Mini
Claude 3.5 Sonnet
Hermes 3 Llama 3.1 405B

LLM Suicide Prompt Jailbreak

Published Jul 1, 2025Analyzed Jul 14, 2025Source: arXiv

Large Language Models (LLMs) employing safety filters designed to prevent generation of content related to self-harm and suicide can be bypassed through multi-step adversarial prompting. By reframing the request as an academic exercise or hypothetical scenario, users can elicit detailed instructions and information that could facilitate self-harm or suicide, despite initially expressing harmful intent. This vulnerability lies in the inadequacy of existing safety filters to consistently…

For Argument's Sake, Show Me How to Harm Myself!': Jailbreaking LLMs in Suicide and Self-Harm Contexts

Evaluated models: Claude 3.7 Sonnet, Gemini 2.0 Flash, GPT-4o

Adaptive Cipher Jailbreak

Published Jun 27, 2025Analyzed Jul 14, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to obfuscation-based jailbreak attacks using the MetaCipher framework. MetaCipher employs a reinforcement learning algorithm to iteratively select from a pool of 21 ciphers to encrypt malicious keywords within prompts, evading standard safety mechanisms that rely on keyword detection. The framework adaptively learns optimal cipher choices to maximize the success rate of the jailbreak, even against LLMs with reasoning capabilities. Successful attacks…

MetaCipher: A General and Extensible Reinforcement Learning Framework for Obfuscation-Based Jailbreak Attacks on Black-Box LLMs

Evaluated models: Claude 3.7 Sonnet, DeepSeek Chat, DeepSeek R1+9 more

Claude 3.7 Sonnet
DeepSeek Chat
DeepSeek R1
Falcon 3 10B Instruct
Gemini 2.0 Flash
Gemini 2.5 Pro
GPT-4o
InternLM 2.5 20B
Llama 3.3 70B Instruct
o1-mini
Qwen 2.5 72B Instruct
QwQ 32B

Variational Jailbreak Inference

Published Jun 27, 2025Analyzed Jul 14, 2025Source: arXiv

VERA, a variational inference framework, enables the generation of diverse and fluent adversarial prompts that bypass safety mechanisms in large language models (LLMs). The attacker model, trained through a variational objective, learns a distribution of prompts likely to elicit harmful responses, effectively jailbreaking the target LLM. This allows for the generation of novel attacks that are not based on pre-existing, manually crafted prompts.

VERA: Variational Inference Framework for Jailbreaking Large Language Models

Evaluated models: Baichuan 2 7B, Gemini Pro, GPT-3.5 Turbo+8 more

Baichuan 2 7B
Gemini Pro
GPT-3.5 Turbo
Llama 2 13B
Llama 2 13B Chat
Llama 2 7B Chat
Llama 3 8B
Mistral 7B
Orca 2 7B
Vicuna 7B
Zephyr 7B

Iterative Semantic Jailbreak

Published Jun 20, 2025Analyzed Jul 14, 2025Source: arXiv

The MIST attack exploits a vulnerability in black-box large language models (LLMs) allowing iterative semantic tuning of prompts to elicit harmful responses. The attack leverages synonym substitution and optimization strategies to bypass safety mechanisms without requiring access to the model's internal parameters or weights. The vulnerability lies in the susceptibility of the LLM to semantically similar prompts that trigger unsafe outputs.

MIST: Jailbreaking Black-box Large Language Models via Iterative Semantic Tuning

Evaluated models: Claude 3.5 Sonnet, GPT-4 Turbo, GPT-4o+3 more

Claude 3.5 Sonnet
GPT-4 Turbo
GPT-4o
GPT-4o Mini
Llama 2 7B Chat
Vicuna 7B v1.5

Doppelgänger Agent Hijack

Published Jun 17, 2025Analyzed Dec 9, 2025Source: arXiv

Large Language Model (LLM) agents are vulnerable to role consistency collapse and privilege escalation via the "Doppelgänger Method," a prompt-based transferable adversarial attack. By exploiting the probabilistic nature of LLM reasoning, an attacker can induce the agent to dissociate from its assigned system persona (defined by system instructions $S$, behavior constraints $B$, and background knowledge $R$) and revert to a default "assistant" or hijacked state. This vulnerability allows…

Doppelgänger Method: Breaking Role Consistency in LLM Agent via Prompt-based Transferable Adversarial Attack

Evaluated models: GPT-4, GPT-4.1, GPT-4.5 Preview+6 more

GPT-4
GPT-4.1
GPT-4.5 Preview
GPT-4o
o3-mini
Gemini 2.5 Flash
HCX-002
HCX-003
HCX-DASH-002

Related research categories