Architecture Components

Fine-tuning Research

Security research involving model fine-tuning processes

126 matching entries

Fine-tuning research results

126 entries

Alignment Override Unlearnable Data

Published Jan 6, 2026Analyzed Mar 8, 2026Source: arXiv

A data poisoning vulnerability in safety-aligned Large Language Models (LLMs) allows attackers to disrupt model fine-tuning via "Disclaimer Injection." By appending or prepending short, legal-style safety or liability disclaimers to ordinary training data, an attacker can reliably trigger the model's internal alignment mechanisms. This forces the model to route the training inputs through specialized safety and refusal pathways rather than standard task-learning layers. Consequently, the model…

Rendering Data Unlearnable by Exploiting LLM Alignment Mechanisms

Evaluated models: GPT-5.1, Llama 3 8B

Quantization Contrastive Jailbreak

Published Jan 6, 2026Analyzed Jan 14, 2026Source: arXiv

A malicious model supply chain vulnerability exists involving a technique termed Adversarial Contrastive Learning (ACL) for Large Language Model (LLM) quantization attacks. This vulnerability allows an attacker to publish a model that appears benign and preserves high utility in full precision (e.g., BF16 or FP32) but exhibits malicious behaviors—such as jailbreak, over-refusal, or advertisement injection—immediately upon zero-shot quantization (e.g., INT8, FP4, or NF4).

Adversarial Contrastive Learning for LLM Quantization Attacks

Evaluated models: Qwen 2.5 1.5B Instruct, Qwen 2.5 3B Instruct, Llama 3.2 1B Instruct+1 more

Qwen 2.5 1.5B Instruct
Qwen 2.5 3B Instruct
Llama 3.2 1B Instruct
Llama 3.2 3B Instruct

Prompt Steers Instrumental Convergence

Published Jan 4, 2026Analyzed Mar 8, 2026Source: arXiv

Open-weight Large Language Models, demonstrated specifically on Qwen3 (4B and 30B-A3B Base, Instruct, and Thinking variants), are vulnerable to unauthorized steerability attacks where minimal inference-time interventions—such as short, pro-instrumental prompt suffixes—reliably elicit dangerous instrumental-convergence behaviors. Because instruction-tuned and "Thinking" models are inherently designed to be highly responsive to steering (authorized steerability), malicious actors can exploit…

Steerability of Instrumental-Convergence Tendencies in LLMs

Evaluated models: Qwen 3 4B Base, Qwen 3 4B Instruct, Qwen 3 4B Thinking+3 more

Qwen 3 4B Base
Qwen 3 4B Instruct
Qwen 3 4B Thinking
Qwen 3 30B-A3B Base
Qwen 3 30B-A3B Instruct
Qwen 3 30B-A3B Thinking

Aggressive Compression Model Theft

Published Jan 3, 2026Analyzed Feb 21, 2026Source: arXiv

Large Language Models (LLMs) hosted on inference servers are vulnerable to high-speed weight exfiltration attacks due to the inherent compressibility of transformer parameters when decompression constraints are relaxed. Adversaries with compromised server access can utilize aggressive lossy compression techniques—specifically additive quantization combined with k-means clustering—to reduce model size by factors of 16x to 100x (e.g., <1 bit per parameter). Unlike standard quantization for…

Aggressive Compression Enables LLM Weight Theft

Evaluated models: Qwen 2 1.5B, Qwen 2 7B, Qwen 2.5 0.5B+2 more

Qwen 2 1.5B
Qwen 2 7B
Qwen 2.5 0.5B
Qwen 2.5 1.5B
Pythia 70M

LoRA Unverified Product Poisoning

Published Jan 2, 2026Analyzed Mar 9, 2026Source: arXiv

In distributed Low-Rank Adaptation (LoRA) fine-tuning systems, a structural verification blind spot exists due to the decoupled aggregation of low-rank matrices. Frameworks typically evaluate and aggregate the $A$ and $B$ matrices independently to reduce computational overhead. A malicious client can exploit this by submitting individually benign $A$ and $B$ matrices that satisfy standard norm-based and similarity-based anomaly detection filters, but whose composite product ($A \times B$)…

Low Rank Comes with Low Security: Gradient Assembly Poisoning Attacks against Distributed LoRA-based LLM Systems

Evaluated models: ChatGLM2 6B, GPT-2 124M, Llama 7B+2 more

ChatGLM2 6B
GPT-2 124M
Llama 7B
Llama 13B
Llama 33B

LLM Anthropomorphic Vulnerabilities

Published Dec 30, 2025Analyzed Jan 14, 2026Source: arXiv

Large Language Models (LLMs) deployed as autonomous agents exhibit "Anthropomorphic Vulnerability Inheritance" (AVI), a vulnerability class where models internalize human psychological failure modes during training. Attackers can bypass security controls and manipulate agent decision-making by exploiting semantic patterns associated with authority bias, artificial urgency, and social proof. Unlike traditional prompt injection which attempts to override system instructions, AVI exploits the…

The Silicon Psyche: Anthropomorphic Vulnerabilities in Large Language Models

Evaluated models: Claude Opus 4.5, Claude Sonnet 4.5, Claude Haiku 4.5+17 more

Claude Opus 4.5
Claude Sonnet 4.5
Claude Haiku 4.5
GPT-5.2 Pro
GPT-5.2 Thinking
o3-preview
o3-mini
Gemini 3 Pro
Gemini 3 Flash
Gemini 3 Deep Think
Llama 4 Scout
Llama 4 Maverick
Llama 3.3 70B
Mistral Large 3
Ministral 3
Devstral 2
DeepSeek V3.2
DeepSeek V3.2 Speciale
DeepSeek R1
Grok 4.1

Abliteration Cripples Math

Published Dec 15, 2025Analyzed Mar 8, 2026Source: arXiv

Open-weight Large Language Models (LLMs) are vulnerable to a white-box safety alignment bypass known as "abliteration" (directional orthogonalization). An attacker with access to the model weights can compute the "refusal direction" in the residual stream activation space by contrasting internal activations between harmful and harmless prompts. By projecting the model's weight matrices to be orthogonal to this single direction (or specific concept cones), the safety alignment is surgically…

Comparative Analysis of LLM Abliteration Methods: A Cross-Architecture Evaluation

Evaluated models: Llama 3.1 8B Instruct, Mistral 7B Instruct v0.3, Qwen 2.5 7B Instruct+13 more

Llama 3.1 8B Instruct
Mistral 7B Instruct v0.3
Qwen 2.5 7B Instruct
Qwen 3 8B
Qwen 3 14B
Yi 1.5 9B Chat
Zephyr 7B Beta
DeepSeek LLM 7B Chat
StableLM 2 12B Chat
Gemma 2 9B IT
Gemma 7B IT
InternLM 2.5 7B Chat
OpenChat 3.5 0106
Phi 3 Small 8k Instruct
Vicuna 7B v1.5
Falcon-Mamba 7B Instruct

Pretrained Leak Jailbreak

Published Dec 14, 2025Analyzed Dec 30, 2025Source: arXiv

Large Language Models (LLMs) finetuned from open-weight pretrained sources inherit adversarial vulnerabilities encoded in the pretrained model's internal representations. An attacker with white-box access to a pretrained model (e.g., Llama-2, Llama-3) can identify linearly separable features in the hidden states that correlate with "transferable" jailbreak prompts. By exploiting these features using a Probe-Guided Projection (PGP) attack, the attacker can optimize adversarial suffixes on the…

One Leak Away: How Pretrained Model Exposure Amplifies Jailbreak Risks in Finetuned LLMs

Evaluated models: Llama 2 7B Chat, Llama 3 8B Instruct, DeepSeek LLM 7B Chat+5 more

Llama 2 7B Chat
Llama 3 8B Instruct
DeepSeek LLM 7B Chat
Gemma 7B IT
Llama 2 13B
Qwen 7B
Vicuna 7B v1.5
Mistral 7B Instruct v0.2

Frontier Multi-Turn Jailbreak

Published Dec 8, 2025Analyzed Jan 14, 2026Source: arXiv

Frontier Large Language Models (LLMs) exhibit a critical vulnerability to automated, adaptive multi-turn adversarial attacks, specifically those utilizing tree-based exploration algorithms (e.g., the TEMPEST framework). Unlike single-turn jailbreaks, this vulnerability exploits the model's inability to maintain safety alignment across extended conversation trajectories. An attacker using an automated agent can dynamically select from multiple adversarial strategies—such as academic framing…

Replicating TEMPEST at Scale: Multi-Turn Adversarial Attacks Against Trillion-Parameter Frontier Models

Evaluated models: Cogito 2.1, DeepSeek V3.1, Gemma 3 12B+7 more

Cogito 2.1
DeepSeek V3.1
Gemma 3 12B
GLM-4.6
GPT-oss 20B
GPT-oss 120B
Kimi K2
Kimi K2 Thinking
MiniMax M2
Mistral Large 3

Guardrail Helpful Mode Jailbreak

Published Nov 27, 2025Analyzed Dec 30, 2025Source: arXiv

A "Helpful Mode" role-confusion vulnerability exists in specific Large Language Model (LLM) safety guardrails, specifically Nemotron-Safety-8B and Granite-Guardian-3.2-5B. These models, designed to act as binary classifiers (outputting "Safe" or "Unsafe") for content moderation, can be manipulated via contextually framed adversarial prompts (e.g., academic research requests, corporate security scenarios, or roleplay) to abandon their classification objective. Instead of blocking the request…

Evaluating the Robustness of Large Language Model Safety Guardrails Against Adversarial Attacks

Evaluated models: Nemotron Safety 8B, Granite Guardian 3.2 5B

Related research categories