Impact

Safety Research

Security concerns affecting AI safety and alignment

56 matching entries out of 798 in this category

Safety research results

56 entries

Single-interaction poisoning of agent memory

Published Aug 24, 2026Analyzed Sep 9, 2026Source: arXiv

A stored interaction can later steer a memory-augmented agent's answer without direct memory-store access. The study evaluates persistent response manipulation in MemoryOS and MemGPT.

InjecMEM: Memory Injection Attack on LLM Agent Memory Systems

Evaluated models: Qwen 2.5 7B Instruct, Qwen 2.5 1.5B Instruct, Qwen 2.5 3B Instruct+3 more

Qwen 2.5 7B Instruct
Qwen 2.5 1.5B Instruct
Qwen 2.5 3B Instruct
Qwen 2.5 14B Instruct
Llama 3.1 8B Instruct
Mistral 7B Instruct v0.3

LLM Judge Manipulation

Published Mar 31, 2026Analyzed Apr 10, 2026Source: arXiv

Generative reward models deployed as LLM-as-a-Judge (LaaJ) evaluators contain a logic bypass vulnerability where superficial "master key" inputs trigger false positive rewards regardless of actual response quality. Instead of evaluating the candidate's output, large judge models are inadvertently triggered by specific token sequences to solve the prompt independently. This allows malicious actors or policy models undergoing reinforcement learning to consistently game the reward signal by…

Security in LLM-as-a-Judge: A Comprehensive SoK

Evaluated models: GPT-4o, o1, Qwen 2.5 72B Instruct+1 more

GPT-4o
o1
Qwen 2.5 72B Instruct
Llama 3 70B Instruct

Autoresearch Adversarial Algorithm Discovery

Published Mar 25, 2026Analyzed Apr 11, 2026Source: arXiv

GPT-OSS-Safeguard-20B and Meta-SecAlign (70B/8B) are vulnerable to white-box adversarial attacks generated by automated algorithmic recombination (specifically the claude_v63, claude_v82, and claude_v53-oss optimizers). These algorithms significantly outperform standard discrete optimization methods (like GCG) by integrating continuous optimization (ADC) with LayerNorm gradient scaling (LSGM), or by merging momentum-smoothed gradients with directional perturbation candidate selection (DPTO)…

Claudini: Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMs

Evaluated models: Llama 2 7B, Llama 3 8B, Qwen 2.5 7B+2 more

Llama 2 7B
Llama 3 8B
Qwen 2.5 7B
Gemma 2 2B
Gemma 7B

Prompt Length Exponential Jailbreak

Published Mar 11, 2026Analyzed Apr 10, 2026Source: arXiv

A vulnerability in safety-aligned Large Language Models (LLMs) allows attackers to achieve an exponentially scaling Attack Success Rate (ASR) for jailbreaks by combining adversarial prompt injection with repeated inference-time sampling. While ASR against un-injected prompts scales polynomially with the number of generated samples ($k$), introducing a long adversarial suffix acts as a strong "misalignment field." This shifts the model's generation distribution into a replica-symmetric ordered…

Jailbreak Scaling Laws for Large Language Models: Polynomial-Exponential Crossover

Evaluated models: Claude Sonnet 4.5 20250929, Claude 3.5 Haiku 20241022, GPT 3.5-turbo-0125+7 more

Claude Sonnet 4.5 20250929
Claude 3.5 Haiku 20241022
GPT 3.5-turbo-0125
GPT-4 0613
GPT-4.5 Preview
Llama 3.2 3B Instruct
Llama 3 8B Instruct
Llama 3 70B Instruct
OLMo 2 0325 32B Instruct
OLMo 3.1 32B Instruct

Visual Memory Multi-Turn Manipulation

Published Feb 17, 2026Analyzed Mar 8, 2026Source: arXiv

Large Vision-Language Models (LVLMs) are vulnerable to Visual Memory Injection (VMI), a stealthy targeted attack targeting multi-turn conversations. An attacker can embed an imperceptible adversarial perturbation ($L_\infty \le 8/255$) into a seemingly benign image. Because the visual input persists in the model's context throughout a multi-turn dialogue, the injected payload remains dormant. By utilizing "benign anchoring" and "context-cycling" during optimization, the attacker ensures the…

Visual Memory Injection Attacks for Multi-Turn Conversations

Evaluated models: Qwen 2.5 VL 7B Instruct, Qwen3-VL 8B Instruct, LLaVA-OneVision 1.5 8B Instruct+2 more

Qwen 2.5 VL 7B Instruct
Qwen3-VL 8B Instruct
LLaVA-OneVision 1.5 8B Instruct
Qwen-SEA-LION v4 8B VL
QoQ-Med3-VL 8B

Dynamic Backtracking Failure

Published Feb 9, 2026Analyzed Feb 22, 2026Source: arXiv

Large Language Models (LLMs) aligned via standard Reinforcement Learning from Human Feedback (RLHF) or Supervised Fine-Tuning (SFT) are vulnerable to "Shallow Safety" bypass attacks, specifically Middle Filling (MF) and Greedy Coordinate Gradient (GCG) attacks. These models frequently rely on refusal mechanisms triggered solely by the initial tokens of a prompt. By embedding malicious instructions after a benign context (prefilling) or utilizing suffix optimization, attackers can induce the…

Reinforcement Learning with Backtracking Feedback

Evaluated models: Llama 3.2 1B, Llama 3.2 3B, Llama 3 8B Instruct+2 more

Llama 3.2 1B
Llama 3.2 3B
Llama 3 8B Instruct
Gemma 2 2B
Gemma 2 9B IT

Agent-as-a-Proxy Monitor Bypass

Published Feb 4, 2026Analyzed Feb 21, 2026Source: arXiv

Hybrid monitoring protocols for Large Language Model (LLM) agents, specifically those that strictly evaluate both Chain-of-Thought (CoT) reasoning and tool usage (such as LlamaFirewall's AlignmentCheck and Extract-and-Evaluate monitors), are vulnerable to "Agent-as-a-Proxy" indirect prompt injection attacks. Unlike standard injections that target the agent, this attack targets the monitoring model itself. By embedding a malicious string—optimized via an algorithm named Parallel-GCG—into…

Bypassing AI Control Protocols via Agent-as-a-Proxy Attacks

Evaluated models: GPT-4o, Llama 3.1 8B, Mistral 7B+1 more

GPT-4o
Llama 3.1 8B
Mistral 7B
Qwen 2.5 7B

Activation Delta Detector Evasion

Published Jan 31, 2026Analyzed Mar 9, 2026Source: arXiv

Activation-delta-based linear probes used for detecting task drift and prompt injections in Large Language Models (LLMs) can be bypassed using universal adversarial suffixes. By utilizing the Greedy Coordinate Gradient (GCG) algorithm, an attacker can generate a single, optimized suffix that simultaneously fools multiple logistic regression classifiers attached to different hidden layers of the LLM. The attack jointly optimizes the suffix tokens by accumulating gradients from the losses of all…

Bypassing Prompt Injection Detectors through Evasive Injections

Evaluated models: Llama 3 8B, Phi-3 8B

Unified Robustness Gap

Published Jan 31, 2026Analyzed Mar 9, 2026Source: arXiv

Text scoring models, including dense retrievers, rerankers, and reward models, are vulnerable to score manipulation attacks via search-based discrete perturbations and content injection. An attacker can systematically modify candidate texts using rudimentary string manipulations, gradient-guided token swaps (e.g., HotFlip), masked language modeling (MLM) swaps, or query/sentence injections to spuriously increase model scores. This structural failure condition allows an irrelevant passage or a…

Unifying Adversarial Robustness and Training Across Text Scoring Models

Evaluated models: E5 BERT-base, Qwen 3 0.6B, Llama 3.2 3B Instruct+2 more

E5 BERT-base
Qwen 3 0.6B
Llama 3.2 3B Instruct
Llama 3.1 8B Instruct
Skywork Reward V2

LLM Router Rerouting

Published Jan 29, 2026Analyzed Mar 8, 2026Source: arXiv

LLM routing systems are vulnerable to adversarial rerouting attacks where malicious triggers prepended to user queries manipulate the router's model-selection mechanism. Because LLM routers function as classifiers evaluating query complexity to balance computational cost and response quality, an attacker can craft adversarial prefixes that distort the query's latent semantic representation. This exploits the router's decision boundaries, forcing the system to misclassify the input and redirect…

RerouteGuard: Understanding and Mitigating Adversarial Risks for LLM Routing

Evaluated models: GPT-4, GPT-4o, GPT-5+2 more

GPT-4
GPT-4o
GPT-5
Llama 3 8B
Mixtral 8x7B

Related research categories