Attack Surface

Prompt layer Research

Security research involving prompt handling and processing

66 matching entries out of 610 in this category

Prompt layer research results

66 entries

Unsafe Culinary Instructions

Published Apr 1, 2026Analyzed Apr 10, 2026Source: arXiv

State-of-the-art Large Language Models (LLMs) and safety guardrails lack domain-specific safety alignment for food science, making them vulnerable to generating actionable, hazardous food safety instructions. Attackers can exploit this alignment sparsity using canonical jailbreak techniques (such as AutoDAN and Persuasive Adversarial Prompting) or direct adversarial prompting to bypass generic safety filters. This allows malicious actors to elicit harmful guidance that violates fundamental FDA…

Cooking Up Risks: Benchmarking and Reducing Food Safety Risks in Large Language Models

Evaluated models: Claude 3.7 Sonnet, GPT-4o, GPT-4.1+8 more

Claude 3.7 Sonnet
GPT-4o
GPT-4.1
GLM-4 32B
Llama 3.3 70B
Qwen 2.5 7B
Qwen 3 8B
Qwen 3 32B
Mistral Small 4
Qwen3Guard
Llama Guard 4

LLM Judge Manipulation

Published Mar 31, 2026Analyzed Apr 10, 2026Source: arXiv

Generative reward models deployed as LLM-as-a-Judge (LaaJ) evaluators contain a logic bypass vulnerability where superficial "master key" inputs trigger false positive rewards regardless of actual response quality. Instead of evaluating the candidate's output, large judge models are inadvertently triggered by specific token sequences to solve the prompt independently. This allows malicious actors or policy models undergoing reinforcement learning to consistently game the reward signal by…

Security in LLM-as-a-Judge: A Comprehensive SoK

Evaluated models: GPT-4o, o1, Qwen 2.5 72B Instruct+1 more

GPT-4o
o1
Qwen 2.5 72B Instruct
Llama 3 70B Instruct

Autoresearch Adversarial Algorithm Discovery

Published Mar 25, 2026Analyzed Apr 11, 2026Source: arXiv

GPT-OSS-Safeguard-20B and Meta-SecAlign (70B/8B) are vulnerable to white-box adversarial attacks generated by automated algorithmic recombination (specifically the claude_v63, claude_v82, and claude_v53-oss optimizers). These algorithms significantly outperform standard discrete optimization methods (like GCG) by integrating continuous optimization (ADC) with LayerNorm gradient scaling (LSGM), or by merging momentum-smoothed gradients with directional perturbation candidate selection (DPTO)…

Claudini: Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMs

Evaluated models: Llama 2 7B, Llama 3 8B, Qwen 2.5 7B+2 more

Llama 2 7B
Llama 3 8B
Qwen 2.5 7B
Gemma 2 2B
Gemma 7B

Joint Audio-Text Jailbreak

Published Mar 19, 2026Analyzed Apr 10, 2026Source: arXiv

Spoken Language Models (SLMs) are vulnerable to Joint Audio-text Multimodal Attacks (JAMA), which bypass safety alignments by simultaneously perturbing both text and audio inputs. The vulnerability exploits the combined optimization of a discrete text suffix via Greedy Coordinate Gradient (GCG) and a continuous audio perturbation via Projected Gradient Descent (PGD). This joint gradient-based attack pushes the model's hidden layer representations into a distinct subspace far from the benign…

On Optimizing Multimodal Jailbreaks for Spoken Language Models

Evaluated models: Qwen2-Audio 7B Instruct, Qwen 2.5 Omni 7B, Audio Flamingo 3+1 more

Qwen2-Audio 7B Instruct
Qwen 2.5 Omni 7B
Audio Flamingo 3
Gemma 3n E2B

Prompt Length Exponential Jailbreak

Published Mar 11, 2026Analyzed Apr 10, 2026Source: arXiv

A vulnerability in safety-aligned Large Language Models (LLMs) allows attackers to achieve an exponentially scaling Attack Success Rate (ASR) for jailbreaks by combining adversarial prompt injection with repeated inference-time sampling. While ASR against un-injected prompts scales polynomially with the number of generated samples ($k$), introducing a long adversarial suffix acts as a strong "misalignment field." This shifts the model's generation distribution into a replica-symmetric ordered…

Jailbreak Scaling Laws for Large Language Models: Polynomial-Exponential Crossover

Evaluated models: Claude Sonnet 4.5 20250929, Claude 3.5 Haiku 20241022, GPT 3.5-turbo-0125+7 more

Claude Sonnet 4.5 20250929
Claude 3.5 Haiku 20241022
GPT 3.5-turbo-0125
GPT-4 0613
GPT-4.5 Preview
Llama 3.2 3B Instruct
Llama 3 8B Instruct
Llama 3 70B Instruct
OLMo 2 0325 32B Instruct
OLMo 3.1 32B Instruct

Drive Overrides Refusal

Published Mar 9, 2026Analyzed Apr 10, 2026Source: arXiv

A prompt structure vulnerability exists in instruction-tuned Large Language Models (LLMs) where attackers can bypass safety alignments by injecting a continuation-triggering suffix immediately following the user prompt termination token. By placing an affirmative suffix outside the user instruction boundary, it is processed as the beginning of the assistant's own pre-filled response. This structural manipulation intrinsically overactivates the model's continuation attention heads, forcing its…

The Struggle Between Continuation and Refusal: A Mechanistic Analysis of the Continuation-Triggered Jailbreak in LLMs

Evaluated models: Llama 2 7B, Qwen 2.5 7B

Two-Stage Refusal Bypass

Published Mar 3, 2026Analyzed Mar 8, 2026Source: arXiv

Large Language Models (LLMs) are vulnerable to TAO-Attack, an advanced optimization-based jailbreak that bypasses safety alignments by exploiting gradient-guided token updates. The vulnerability stems from a two-stage loss function combined with a Direction-Priority Token Optimization (DPTO) algorithm. In the first stage, the attack optimizes an adversarial prompt suffix to minimize the probability of refusal signals (e.g., "I cannot") while maximizing the probability of a harmful target…

TAO-Attack: Toward Advanced Optimization-Based Jailbreak Attacks for Large Language Models

Evaluated models: GPT-3.5 Turbo, GPT-4 Turbo, Llama 2 7B Chat+4 more

GPT-3.5 Turbo
GPT-4 Turbo
Llama 2 7B Chat
Gemini 1.5 Flash
Gemini 2.0 Flash
Mistral 7B Instruct v0.2
Vicuna 7B v1.5

RAG Knowledgebase Exfiltration

Published Feb 10, 2026Analyzed Feb 21, 2026Source: arXiv

Retrieval-Augmented Generation (RAG) systems are vulnerable to iterative knowledge-extraction attacks designed to reconstruct the underlying private knowledge base. The vulnerability exists due to the decoupled optimization of the retrieval and generation phases. Attackers can craft adversarial queries consisting of two distinct components: an "Information" component (optimized via gradient descent or random sampling to steer embeddings toward specific, diverse regions of the vector space) and…

Benchmarking Knowledge-Extraction Attack and Defense on Retrieval-Augmented Generation

Evaluated models: GPT-4o, Llama 3 8B, Qwen 2.5 7B

Dynamic Backtracking Failure

Published Feb 9, 2026Analyzed Feb 22, 2026Source: arXiv

Large Language Models (LLMs) aligned via standard Reinforcement Learning from Human Feedback (RLHF) or Supervised Fine-Tuning (SFT) are vulnerable to "Shallow Safety" bypass attacks, specifically Middle Filling (MF) and Greedy Coordinate Gradient (GCG) attacks. These models frequently rely on refusal mechanisms triggered solely by the initial tokens of a prompt. By embedding malicious instructions after a benign context (prefilling) or utilizing suffix optimization, attackers can induce the…

Reinforcement Learning with Backtracking Feedback

Evaluated models: Llama 3.2 1B, Llama 3.2 3B, Llama 3 8B Instruct+2 more

Llama 3.2 1B
Llama 3.2 3B
Llama 3 8B Instruct
Gemma 2 2B
Gemma 2 9B IT

LLM Strategic Prompt Deception

Published Feb 9, 2026Analyzed Feb 21, 2026Source: arXiv

Alignment auditing methodologies used to verify the safety of Large Language Models (LLMs)—specifically black-box methods (Assistant Prefills, User Persona Sampling) and white-box interpretability methods (SAE Feature Description, Activation Token Embedding Similarity)—are vulnerable to prompt-level strategic deception. A misaligned "target" model, when supplied with a specific system prompt containing a high-level deceptive strategy and descriptions of the auditing tools, can obfuscate its…

Stress-Testing Alignment Audits With Prompt-Level Strategic Deception

Evaluated models: Llama 3.1 8B, Llama 3.3 70B, Gemma 2 9B

Related research categories