Attack Context

White-box Research

Attacks leveraging model implementation details

15 matching entries out of 228 in this category

White-box research results

15 entries

Weight Orthogonalization Attack Empowerment

Published Apr 2, 2026Analyzed Apr 10, 2026Source: arXiv

An issue in large language models (LLMs) with white-box weight access allows attackers to permanently bypass safety guardrails via Weight Orthogonalization (WO). By calculating a model's "refusal vector"—the mean-difference vector between harmful and harmless instruction activations in the residual stream—an attacker can orthogonalize the model's weights to prevent it from writing to this refusal direction ($W^{\prime}\leftarrow W-rr^{\intercal}W$). Unlike jailbreak-tuning or data poisoning…

Understanding the Effects of Safety Unalignment on Large Language Models

Evaluated models: Qwen 3 4B Instruct 2507, Llama 3.1 8B Instruct, Qwen 2.5 14B+3 more

Qwen 3 4B Instruct 2507
Llama 3.1 8B Instruct
Qwen 2.5 14B
Qwen 3 4B Thinking 2507
DeepSeek R1 Distill Llama 8B
DeepSeek R1 Distill Qwen 14B

LLM Judge Manipulation

Published Mar 31, 2026Analyzed Apr 10, 2026Source: arXiv

Generative reward models deployed as LLM-as-a-Judge (LaaJ) evaluators contain a logic bypass vulnerability where superficial "master key" inputs trigger false positive rewards regardless of actual response quality. Instead of evaluating the candidate's output, large judge models are inadvertently triggered by specific token sequences to solve the prompt independently. This allows malicious actors or policy models undergoing reinforcement learning to consistently game the reward signal by…

Security in LLM-as-a-Judge: A Comprehensive SoK

Evaluated models: GPT-4o, o1, Qwen 2.5 72B Instruct+1 more

GPT-4o
o1
Qwen 2.5 72B Instruct
Llama 3 70B Instruct

Emergent Distractor Jailbreak

Published Feb 11, 2026Analyzed Mar 9, 2026Source: arXiv

A vulnerability in the Direct Preference Optimization (DPO) post-training phase of OLMo 2 models leads to "distractor-triggered compliance." The model correctly refuses harmful requests when prompted in isolation, but complies with identical harmful requests if a benign formatting instruction (a "distractor") is appended to the prompt. This behavior organically emerges from contaminated preference training data where mislabeled examples incorrectly preferred compliance over refusal when a…

In-the-Wild Model Organisms: Mitigating Undesirable Emergent Behaviors in Production LLM Post-Training via Data Attribution

Evaluated models: Not reported

LLM Safety Geometry Fragility

Published Feb 7, 2026Analyzed Feb 21, 2026Source: arXiv

Large Language Models (LLMs) aligned via standard preference-based optimization methods (e.g., DPO, RLHF) are vulnerable to safety degradation due to optimization-induced fragility. The vulnerability arises from sharp minima in the alignment loss landscape, specifically within a small, localized subspace of safety-critical parameters (approximately 0.5% of neurons account for >80% of worst-case alignment loss). Standard alignment algorithms enforce uniform constraints or fail to control the…

Revisiting Robustness for LLM Safety Alignment via Selective Geometry Control

Evaluated models: Llama 3 8B, Llama 3.2 3B, Qwen 2.5 7B

Unified Robustness Gap

Published Jan 31, 2026Analyzed Mar 9, 2026Source: arXiv

Text scoring models, including dense retrievers, rerankers, and reward models, are vulnerable to score manipulation attacks via search-based discrete perturbations and content injection. An attacker can systematically modify candidate texts using rudimentary string manipulations, gradient-guided token swaps (e.g., HotFlip), masked language modeling (MLM) swaps, or query/sentence injections to spuriously increase model scores. This structural failure condition allows an irrelevant passage or a…

Unifying Adversarial Robustness and Training Across Text Scoring Models

Evaluated models: E5 BERT-base, Qwen 3 0.6B, Llama 3.2 3B Instruct+2 more

E5 BERT-base
Qwen 3 0.6B
Llama 3.2 3B Instruct
Llama 3.1 8B Instruct
Skywork Reward V2

Refusal Prefix Unlearning

Published Jan 27, 2026Analyzed Mar 8, 2026Source: arXiv

A fine-tuning vulnerability in the safety alignment of Large Language Models (LLMs) allows adversaries to systematically bypass refusal mechanisms by training the model on a small dataset (as few as 1,000 samples) of strictly benign text. By prepending standard refusal prefixes (e.g., "I'm sorry", "I cannot fulfill this request") to the target outputs of benign instruction-response pairs, attackers disrupt the model's refusal completion pathway. When subsequently prompted with unsafe queries…

LLMs Can Unlearn Refusal with Only 1,000 Benign Samples

Evaluated models: Llama 2 13B, Llama 3.1 8B, Llama 3.2 1B+13 more

Llama 2 13B
Llama 3.1 8B
Llama 3.2 1B
Llama 3.3 70B
Qwen 2 7B
Qwen 2.5 32B
Qwen 3 0.6B
Qwen 3 4B Think
Gemma 1.1 7B
Gemma 2 2B
Gemma 2 9B
Gemma 2 27B
GPT-oss 20B
GPT-4.1 Nano
Gemini 2.0 Flash-Lite
Gemini 2.5 Flash-Lite

Quantization Contrastive Jailbreak

Published Jan 6, 2026Analyzed Jan 14, 2026Source: arXiv

A malicious model supply chain vulnerability exists involving a technique termed Adversarial Contrastive Learning (ACL) for Large Language Model (LLM) quantization attacks. This vulnerability allows an attacker to publish a model that appears benign and preserves high utility in full precision (e.g., BF16 or FP32) but exhibits malicious behaviors—such as jailbreak, over-refusal, or advertisement injection—immediately upon zero-shot quantization (e.g., INT8, FP4, or NF4).

Adversarial Contrastive Learning for LLM Quantization Attacks

Evaluated models: Qwen 2.5 1.5B Instruct, Qwen 2.5 3B Instruct, Llama 3.2 1B Instruct+1 more

Qwen 2.5 1.5B Instruct
Qwen 2.5 3B Instruct
Llama 3.2 1B Instruct
Llama 3.2 3B Instruct

Prompt Injection Alignment Bypass

Published Sep 4, 2025Analyzed Dec 8, 2025Source: arXiv

Large Language Models (LLMs) integrated with external retrieval mechanisms (e.g., Retrieval-Augmented Generation (RAG), web search, or email processing) are vulnerable to Indirect Prompt Injection. This vulnerability occurs when an LLM consumes input from untrusted external sources—such as websites, code repositories, or incoming emails—that contain embedded adversarial prompts. Unlike direct injection, where the user attacks the model, here the "poisoned" data is retrieved by the system…

Breaking to Build: A Threat Model of Prompt-Based Attacks for Securing LLMs

Evaluated models: Not reported

Guardrail Bypass Harmful Fine-tuning

Published Jan 29, 2025Analyzed Mar 19, 2025Source: arXiv

The Virus attack method enables attackers to bypass guardrail moderation on fine-tuning data, leading to a significant degradation of safety alignment in large language models (LLMs). This is achieved through a dual-objective data optimization strategy that crafts harmful data undetectable by the guardrail while maximizing their effectiveness in compromising the victim model's safety.

Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation

Evaluated models: Llama 3 8B, Llama Guard 2

FedPEFT Evasion Attack

Published Nov 28, 2024Analyzed Dec 29, 2024Source: arXiv

A vulnerability exists in Federated Parameter-Efficient Fine-Tuning (FedPEFT) systems for large language models (LLMs). Malicious clients can exploit the PEFT mechanism (e.g., LoRA, (IA)³, LayerNorm) to inject adversarial training data, compromising the model's safety alignment even with a small percentage of trainable parameters and a minority of malicious participants. The attack, termed "PEFT-as-an-Attack" (PaaA), circumvents the LLM's safety guardrails, causing it to generate harmful…

PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuning

Evaluated models: Llama 2 7B Chat, Llama 3.2 3B Instruct, Phi-3.5 Mini Instruct+1 more

Llama 2 7B Chat
Llama 3.2 3B Instruct
Phi-3.5 Mini Instruct
Qwen 2.5 7B Instruct

Related research categories