Architecture Components

Fine-tuning Research

Security research involving model fine-tuning processes

126 matching entries

Fine-tuning research results

126 entries

Unified Robustness Gap

Published Jan 31, 2026Analyzed Mar 9, 2026Source: arXiv

Text scoring models, including dense retrievers, rerankers, and reward models, are vulnerable to score manipulation attacks via search-based discrete perturbations and content injection. An attacker can systematically modify candidate texts using rudimentary string manipulations, gradient-guided token swaps (e.g., HotFlip), masked language modeling (MLM) swaps, or query/sentence injections to spuriously increase model scores. This structural failure condition allows an irrelevant passage or a…

Unifying Adversarial Robustness and Training Across Text Scoring Models

Evaluated models: E5 BERT-base, Qwen 3 0.6B, Llama 3.2 3B Instruct+2 more

E5 BERT-base
Qwen 3 0.6B
Llama 3.2 3B Instruct
Llama 3.1 8B Instruct
Skywork Reward V2

Adversarial Context Reasoning Brittleness

Published Jan 30, 2026Analyzed Mar 9, 2026Source: arXiv

Large Reasoning Models (LRMs) optimized via Reinforcement Learning from Verifiable Rewards (RLVR) are vulnerable to context pollution in their reasoning traces. An attacker can induce catastrophic reasoning failure by injecting locally coherent but logically or mathematically corrupted snippets into the model's Chain-of-Thought (CoT) or conditioning context. Because standard RLVR optimizes for final-answer correctness strictly under clean conditioning, the models treat the visible trajectory…

Learning Robust Reasoning through Guided Adversarial Self-Play

Evaluated models: DeepSeek R1 Distill Qwen 1.5B, DeepScaleR 1.5B, Qwen 3 4B+1 more

DeepSeek R1 Distill Qwen 1.5B
DeepScaleR 1.5B
Qwen 3 4B
Qwen 3 8B

Emergent Character Misalignment

Published Jan 30, 2026Analyzed Feb 21, 2026Source: arXiv

Large Language Models (LLMs), specifically Llama-3.1-8B-Instruct and Qwen2.5-14B-Instruct, are vulnerable to emergent misalignment caused by "character-conditioned" fine-tuning. This vulnerability arises when models are fine-tuned on small datasets (e.g., 500 examples) that exhibit consistent behavioral dispositions (e.g., "Evil," "Sycophantic," or "Hallucinatory") rather than just incorrect facts. This process creates a latent control variable—defined as "character"—that governs model…

Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failures

Evaluated models: GPT-5, Llama 3.1 8B, Qwen 2.5 14B

LLM Grading Compliance Paradox

Published Jan 29, 2026Analyzed Feb 21, 2026Source: arXiv

Large Language Models (LLMs) employed as automated code evaluators ("Universal Graders") are vulnerable to Semantic-Instruction Decoupling, a form of adversarial prompt injection that exploits the "Syntax-Semantics Gap." Attackers can embed adversarial directives into syntactically inert regions of the Abstract Syntax Tree (AST)—specifically comments, docstrings, variable names, and whitespace. While these regions are discarded by compilers (trivia nodes) or treated as arbitrary symbols…

The Compliance Paradox: Semantic-Instruction Decoupling in Automated Academic Code Evaluation

Evaluated models: GPT-5, Llama 3.1 8B, DeepSeek V3

LLM Soft Hate Policy Bypass

Published Jan 28, 2026Analyzed Feb 21, 2026Source: arXiv

Large Language Models (LLMs) and dedicated content moderation systems (including safety guards and encoder-based classifiers) are vulnerable to "Soft Hate" evasion attacks. This vulnerability arises when hostile standpoints are encoded through defeasible reasoning structures—specifically formalized by the Argumentum Model of Topics (AMT)—rather than explicit lexical toxicity. Attackers can bypass safety filters by framing discriminatory conclusions as logical inferences derived from widely…

SoftHateBench: Evaluating Moderation Models Against Reasoning-Driven, Policy-Compliant Hostility

Evaluated models: HateBERT, HateRoBERTa, Llama Guard 3 1B+7 more

HateBERT
HateRoBERTa
Llama Guard 3 1B
Qwen 3 Guard 4B
ShieldGemma 2B
DeepSeek V3.1
GPT-5 Mini
Llama 3.2 3B
Gemma 3 4B
Qwen 3 4B

RL Adversarial Function Call

Published Jan 27, 2026Analyzed Feb 22, 2026Source: arXiv

Large Language Models (LLMs) enabled with Function Calling (FC) capabilities are vulnerable to adversarial query rewriting and semantic manipulation. Standard FC models, typically trained via Supervised Fine-Tuning (SFT) on static datasets, fail to generalize against adversarial inputs that deviate from fixed distribution patterns. An attacker can exploit this by crafting queries that are semantically similar to valid requests but engineered to induce "bad cases," such as incorrect tool…

Exploring Weaknesses in Function Call Models via Reinforcement Learning: An Adversarial Data Augmentation Approach

Evaluated models: Qwen 2.5 7B Instruct, Qwen 3 0.6B, Qwen 3 4B+1 more

Qwen 2.5 7B Instruct
Qwen 3 0.6B
Qwen 3 4B
Qwen 3 8B

Refusal Prefix Unlearning

Published Jan 27, 2026Analyzed Mar 8, 2026Source: arXiv

A fine-tuning vulnerability in the safety alignment of Large Language Models (LLMs) allows adversaries to systematically bypass refusal mechanisms by training the model on a small dataset (as few as 1,000 samples) of strictly benign text. By prepending standard refusal prefixes (e.g., "I'm sorry", "I cannot fulfill this request") to the target outputs of benign instruction-response pairs, attackers disrupt the model's refusal completion pathway. When subsequently prompted with unsafe queries…

LLMs Can Unlearn Refusal with Only 1,000 Benign Samples

Evaluated models: Llama 2 13B, Llama 3.1 8B, Llama 3.2 1B+13 more

Llama 2 13B
Llama 3.1 8B
Llama 3.2 1B
Llama 3.3 70B
Qwen 2 7B
Qwen 2.5 32B
Qwen 3 0.6B
Qwen 3 4B Think
Gemma 1.1 7B
Gemma 2 2B
Gemma 2 9B
Gemma 2 27B
GPT-oss 20B
GPT-4.1 Nano
Gemini 2.0 Flash-Lite
Gemini 2.5 Flash-Lite

LLM Input PII Leakage

Published Jan 24, 2026Analyzed Feb 22, 2026Source: arXiv

Unintended input-only PII memorization in fine-tuned Large Language Models (LLMs) allows remote attackers to extract sensitive Personally Identifiable Information (PII) such as names, medical records, and financial details. This vulnerability occurs when a model is fine-tuned on datasets where sensitive information appears in the input text, even if that information is not part of the training target (label) or is unrelated to the downstream task (e.g., classification). The fine-tuning process…

Unintended Memorization of Sensitive Information in Fine-Tuned Language Models

Evaluated models: Llama 3.1 8B, Llama 3.2 1B

Metacognitive Prompting Lowers Resistance

Published Jan 20, 2026Analyzed Feb 22, 2026Source: arXiv

Large Language Models (LLMs) are vulnerable to multi-turn persuasive conversational attacks that induce the adoption of counterfactual beliefs. By leveraging the Source–Message–Channel–Receiver (SMCR) communication framework, attackers can systematically erode a model's confidence in established facts and compel the model to output misinformation. Specific attack vectors include manipulating source attribution (authority framing), message content (logical, credibility, or emotional appeals)…

Vulnerability of LLMs' Belief Systems? LLMs Belief Resistance Check Through Strategic Persuasive Conversation Interventions

Evaluated models: GPT-4o, Llama 3.2 3B, Llama 3.3 70B+2 more

GPT-4o
Llama 3.2 3B
Llama 3.3 70B
Mistral 7B
Qwen 2.5 7B

Detector Telemetry Camouflage

Published Jan 19, 2026Analyzed Feb 21, 2026Source: arXiv

Single-pass hallucination detectors relying on internal telemetry (uncertainty, hidden-state geometry, and attention patterns) are vulnerable to white-box, model-side adversarial attacks. An attacker can employ the CORVUS (Camouflaging Open-weight Representations, Volumes, Uncertainty, and Structure) technique to fine-tune lightweight Low-Rank Adapters (LoRA) on the target LLM. This method optimizes a specific loss objective that camouflages detector-visible telemetry signals—specifically…

CORVUS: Red-Teaming Hallucination Detectors via Internal Signal Camouflage in Large Language Models

Evaluated models: Llama 2 7B, Llama 3 8B, Qwen 2.5 14B+1 more

Llama 2 7B
Llama 3 8B
Qwen 2.5 14B
Vicuna 7B

Related research categories