Architecture Components

Fine-tuning Research

Security research involving model fine-tuning processes

24 matching entries out of 126 in this category

Fine-tuning research results

24 entries

Weight Orthogonalization Attack Empowerment

Published Apr 2, 2026Analyzed Apr 10, 2026Source: arXiv

An issue in large language models (LLMs) with white-box weight access allows attackers to permanently bypass safety guardrails via Weight Orthogonalization (WO). By calculating a model's "refusal vector"—the mean-difference vector between harmful and harmless instruction activations in the residual stream—an attacker can orthogonalize the model's weights to prevent it from writing to this refusal direction ($W^{\prime}\leftarrow W-rr^{\intercal}W$). Unlike jailbreak-tuning or data poisoning…

Understanding the Effects of Safety Unalignment on Large Language Models

Evaluated models: Qwen 3 4B Instruct 2507, Llama 3.1 8B Instruct, Qwen 2.5 14B+3 more

Qwen 3 4B Instruct 2507
Llama 3.1 8B Instruct
Qwen 2.5 14B
Qwen 3 4B Thinking 2507
DeepSeek R1 Distill Llama 8B
DeepSeek R1 Distill Qwen 14B

LLM Judge Manipulation

Published Mar 31, 2026Analyzed Apr 10, 2026Source: arXiv

Generative reward models deployed as LLM-as-a-Judge (LaaJ) evaluators contain a logic bypass vulnerability where superficial "master key" inputs trigger false positive rewards regardless of actual response quality. Instead of evaluating the candidate's output, large judge models are inadvertently triggered by specific token sequences to solve the prompt independently. This allows malicious actors or policy models undergoing reinforcement learning to consistently game the reward signal by…

Security in LLM-as-a-Judge: A Comprehensive SoK

Evaluated models: GPT-4o, o1, Qwen 2.5 72B Instruct+1 more

GPT-4o
o1
Qwen 2.5 72B Instruct
Llama 3 70B Instruct

Zero-Tax Fine-Tune Bypass

Published Mar 30, 2026Analyzed Apr 10, 2026Source: arXiv

An adversarial fine-tuning vulnerability exists in LLMs protected by text-based safety classifiers (such as Anthropic's Constitutional Classifiers). By utilizing a two-stage curriculum learning combined with hybrid RL+SFT (GRPO), an attacker can fine-tune a model to communicate using a minimal substitution cipher (replacing only 7-8 high-frequency characters) disguised within benign technical templates (e.g., forensic logs with 0x prefixes). This "Trojan-Speak" methodology bypasses text-level…

Trojan-Speak: Bypassing Constitutional Classifiers with No Jailbreak Tax via Adversarial Finetuning

Evaluated models: Claude Haiku 4.5, Qwen 3 4B, Qwen 3 8B+2 more

Claude Haiku 4.5
Qwen 3 4B
Qwen 3 8B
Qwen 3 14B
Qwen 3 32B

LLM Steganographic Covert Channel

Published Mar 9, 2026Analyzed Apr 10, 2026Source: arXiv

A malicious finetuning vulnerability exists in Large Language Models (LLMs) that process zero-width Unicode characters. An attacker can bypass training-data moderation filters and inference-time safety guardrails by finetuning the model to decode and encode invisible-character steganography. By injecting target malicious interactions encoded in a base-4 representation of zero-width characters alongside benign plaintext cover text during supervised finetuning (SFT), the model learns to process…

Invisible Safety Threat: Malicious Finetuning for LLM via Steganography

Evaluated models: GPT-4.1, Llama 3.3 70B Instruct, Phi-4+1 more

GPT-4.1
Llama 3.3 70B Instruct
Phi-4
Mistral Small 24B Base 2501

Emergent Distractor Jailbreak

Published Feb 11, 2026Analyzed Mar 9, 2026Source: arXiv

A vulnerability in the Direct Preference Optimization (DPO) post-training phase of OLMo 2 models leads to "distractor-triggered compliance." The model correctly refuses harmful requests when prompted in isolation, but complies with identical harmful requests if a benign formatting instruction (a "distractor") is appended to the prompt. This behavior organically emerges from contaminated preference training data where mislabeled examples incorrectly preferred compliance over refusal when a…

In-the-Wild Model Organisms: Mitigating Undesirable Emergent Behaviors in Production LLM Post-Training via Data Attribution

Evaluated models: Not reported

LLM Safety Geometry Fragility

Published Feb 7, 2026Analyzed Feb 21, 2026Source: arXiv

Large Language Models (LLMs) aligned via standard preference-based optimization methods (e.g., DPO, RLHF) are vulnerable to safety degradation due to optimization-induced fragility. The vulnerability arises from sharp minima in the alignment loss landscape, specifically within a small, localized subspace of safety-critical parameters (approximately 0.5% of neurons account for >80% of worst-case alignment loss). Standard alignment algorithms enforce uniform constraints or fail to control the…

Revisiting Robustness for LLM Safety Alignment via Selective Geometry Control

Evaluated models: Llama 3 8B, Llama 3.2 3B, Qwen 2.5 7B

Unified Robustness Gap

Published Jan 31, 2026Analyzed Mar 9, 2026Source: arXiv

Text scoring models, including dense retrievers, rerankers, and reward models, are vulnerable to score manipulation attacks via search-based discrete perturbations and content injection. An attacker can systematically modify candidate texts using rudimentary string manipulations, gradient-guided token swaps (e.g., HotFlip), masked language modeling (MLM) swaps, or query/sentence injections to spuriously increase model scores. This structural failure condition allows an irrelevant passage or a…

Unifying Adversarial Robustness and Training Across Text Scoring Models

Evaluated models: E5 BERT-base, Qwen 3 0.6B, Llama 3.2 3B Instruct+2 more

E5 BERT-base
Qwen 3 0.6B
Llama 3.2 3B Instruct
Llama 3.1 8B Instruct
Skywork Reward V2

Emergent Character Misalignment

Published Jan 30, 2026Analyzed Feb 21, 2026Source: arXiv

Large Language Models (LLMs), specifically Llama-3.1-8B-Instruct and Qwen2.5-14B-Instruct, are vulnerable to emergent misalignment caused by "character-conditioned" fine-tuning. This vulnerability arises when models are fine-tuned on small datasets (e.g., 500 examples) that exhibit consistent behavioral dispositions (e.g., "Evil," "Sycophantic," or "Hallucinatory") rather than just incorrect facts. This process creates a latent control variable—defined as "character"—that governs model…

Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failures

Evaluated models: GPT-5, Llama 3.1 8B, Qwen 2.5 14B

Refusal Prefix Unlearning

Published Jan 27, 2026Analyzed Mar 8, 2026Source: arXiv

A fine-tuning vulnerability in the safety alignment of Large Language Models (LLMs) allows adversaries to systematically bypass refusal mechanisms by training the model on a small dataset (as few as 1,000 samples) of strictly benign text. By prepending standard refusal prefixes (e.g., "I'm sorry", "I cannot fulfill this request") to the target outputs of benign instruction-response pairs, attackers disrupt the model's refusal completion pathway. When subsequently prompted with unsafe queries…

LLMs Can Unlearn Refusal with Only 1,000 Benign Samples

Evaluated models: Llama 2 13B, Llama 3.1 8B, Llama 3.2 1B+13 more

Llama 2 13B
Llama 3.1 8B
Llama 3.2 1B
Llama 3.3 70B
Qwen 2 7B
Qwen 2.5 32B
Qwen 3 0.6B
Qwen 3 4B Think
Gemma 1.1 7B
Gemma 2 2B
Gemma 2 9B
Gemma 2 27B
GPT-oss 20B
GPT-4.1 Nano
Gemini 2.0 Flash-Lite
Gemini 2.5 Flash-Lite

Benign Praise Jailbreak

Published Jan 18, 2026Analyzed Feb 20, 2026Source: arXiv

A vulnerability exists in Large Language Model (LLM) Fine-tuning-as-a-Service (FaaS) platforms that allows attackers to bypass safety alignment and moderation filters via a "TrojanPraise" benign fine-tuning attack. The attack exploits the decoupling of an LLM's internal representation of harmful queries into "knowledge" (semantic understanding) and "attitude" (safety refusal). The attacker constructs a fine-tuning dataset containing three specific components: (1) a novel nonsense word (e.g…

TrojanPraise: Jailbreak LLMs via Benign Fine-Tuning

Evaluated models: GPT-3.5, GPT-4o, Llama 2 7B+4 more

GPT-3.5
GPT-4o
Llama 2 7B
Llama 3 8B
Llama 3.1 8B
Mistral 7B
Qwen 2.5 3B

Related research categories