Architecture Components

Fine-tuning Research

Security research involving model fine-tuning processes

123 matching entries

Fine-tuning research results

123 entries

LLM Robot JSON Backdoor

Published 4/1/2026

Analyzed 4/10/2026

A supply-chain vulnerability in LLM-mediated robotic control systems allows attackers to execute unauthorized physical actions via structured backdoor attacks embedded in LoRA adapters. By poisoning the fine-tuning dataset to map specific natural-language trigger phrases directly to malicious, syntactically valid JSON control commands (structured-output poisoning), the backdoor bypasses natural-language reasoning layers and propagates deterministically to downstream robotic middleware (e.g…

From Prompt to Physical Action: Structured Backdoor Attacks on LLM-Mediated Robotic Control Systems

Evaluated models: Llama 3.1 8B Instruct, Gemma 2 9B IT, DeepSeek R1 Distill Llama 8B +2 more

Source: arXiv

Weight Orthogonalization Attack Empowerment

Published 4/1/2026

Analyzed 4/10/2026

An issue in large language models (LLMs) with white-box weight access allows attackers to permanently bypass safety guardrails via Weight Orthogonalization (WO). By calculating a model's "refusal vector"—the mean-difference vector between harmful and harmless instruction activations in the residual stream—an attacker can orthogonalize the model's weights to prevent it from writing to this refusal direction ($W^{\prime}\leftarrow W-rr^{\intercal}W$). Unlike jailbreak-tuning or data poisoning…

Understanding the Effects of Safety Unalignment on Large Language Models

Evaluated models: Qwen 3 4B Instruct 2507, Llama 3.1 8B Instruct, Qwen 2.5 14B +3 more

Source: arXiv

Colluding LoRA Emergent Harm

Published 3/1/2026

Analyzed 4/10/2026

A compositional vulnerability in modular Large Language Models (LLMs) allows attackers to bypass safety alignment by distributing malicious weight updates across multiple Parameter-Efficient Fine-Tuning (PEFT) adapters (e.g., LoRA). The malicious adapters are anchored to valid functional subspaces (e.g., math, coding) and exhibit benign behavior when evaluated in isolation, successfully evading standard unit-centric safety scans and static weight-space defenses. However, when a user linearly…

Colluding LoRA: A Composite Attack on LLM Safety Alignment

Evaluated models: Llama 3 8B, Qwen 2.5 7B, Gemma 2 2B

Source: arXiv

LLM Judge Manipulation

Published 3/1/2026

Analyzed 4/10/2026

Generative reward models deployed as LLM-as-a-Judge (LaaJ) evaluators contain a logic bypass vulnerability where superficial "master key" inputs trigger false positive rewards regardless of actual response quality. Instead of evaluating the candidate's output, large judge models are inadvertently triggered by specific token sequences to solve the prompt independently. This allows malicious actors or policy models undergoing reinforcement learning to consistently game the reward signal by…

Security in LLM-as-a-Judge: A Comprehensive SoK

Evaluated models: GPT-4o, o1, Qwen 2.5 72B Instruct +1 more

Source: arXiv

LLM Steganographic Covert Channel

Published 3/1/2026

Analyzed 4/10/2026

A malicious finetuning vulnerability exists in Large Language Models (LLMs) that process zero-width Unicode characters. An attacker can bypass training-data moderation filters and inference-time safety guardrails by finetuning the model to decode and encode invisible-character steganography. By injecting target malicious interactions encoded in a base-4 representation of zero-width characters alongside benign plaintext cover text during supervised finetuning (SFT), the model learns to process…

Invisible Safety Threat: Malicious Finetuning for LLM via Steganography

Evaluated models: GPT-4.1, Llama 3.3 70B Instruct, Phi-4 +1 more

Source: arXiv

Zero-Tax Fine-Tune Bypass

Published 3/1/2026

Analyzed 4/10/2026

An adversarial fine-tuning vulnerability exists in LLMs protected by text-based safety classifiers (such as Anthropic's Constitutional Classifiers). By utilizing a two-stage curriculum learning combined with hybrid RL+SFT (GRPO), an attacker can fine-tune a model to communicate using a minimal substitution cipher (replacing only 7-8 high-frequency characters) disguised within benign technical templates (e.g., forensic logs with 0x prefixes). This "Trojan-Speak" methodology bypasses text-level…

Trojan-Speak: Bypassing Constitutional Classifiers with No Jailbreak Tax via Adversarial Finetuning

Evaluated models: Claude Haiku 4.5, Qwen 3 4B, Qwen 3 8B +2 more

Source: arXiv

Adversarial Context Reasoning Brittleness

Published 2/1/2026

Analyzed 3/9/2026

Large Reasoning Models (LRMs) optimized via Reinforcement Learning from Verifiable Rewards (RLVR) are vulnerable to context pollution in their reasoning traces. An attacker can induce catastrophic reasoning failure by injecting locally coherent but logically or mathematically corrupted snippets into the model's Chain-of-Thought (CoT) or conditioning context. Because standard RLVR optimizes for final-answer correctness strictly under clean conditioning, the models treat the visible trajectory…

Learning Robust Reasoning through Guided Adversarial Self-Play

Evaluated models: DeepSeek R1 Distill Qwen 1.5B, DeepScaleR 1.5B, Qwen 3 4B +1 more

Source: arXiv

Chunky Post-Training Miscalibration

Published 2/1/2026

Analyzed 2/21/2026

Large Language Models (LLMs) exhibit a vulnerability termed "chunky post-training," where the model learns spurious correlations between incidental prompt features (e.g., formatting styles, specific vocabulary, sentence structure) and specific behavioral modes (e.g., refusal, code generation, rebuttal) present in distinct chunks of post-training data. This results in behavioral mis-routing during inference, where benign inputs sharing surface-level features with restricted or specialized…

Chunky Post-Training: Data Driven Failures of Generalization

Evaluated models: Claude Haiku 4.5, Claude Sonnet 4.5, Claude Opus 4.5 +5 more

Source: arXiv

Discrete Token Subversion

Published 2/1/2026

Analyzed 3/9/2026

Discrete image tokenizers are vulnerable to unsupervised embedding-space adversarial attacks. Attackers can apply $\ell_p$-bounded perturbations to an input image to maximize the $\ell_2$ distance of the pre-quantization continuous embeddings produced by the tokenizer's vision encoder. This forces the vector quantizer to cross discrete cell boundaries and assign incorrect codebook vectors, fundamentally altering the resulting token sequence. Because the attack targets the pre-quantization…

On the Adversarial Robustness of Discrete Image Tokenizers

Evaluated models: Llama 2 7B

Source: arXiv

Dynamic Backtracking Failure

Published 2/1/2026

Analyzed 2/22/2026

Large Language Models (LLMs) aligned via standard Reinforcement Learning from Human Feedback (RLHF) or Supervised Fine-Tuning (SFT) are vulnerable to "Shallow Safety" bypass attacks, specifically Middle Filling (MF) and Greedy Coordinate Gradient (GCG) attacks. These models frequently rely on refusal mechanisms triggered solely by the initial tokens of a prompt. By embedding malicious instructions after a benign context (prefilling) or utilizing suffix optimization, attackers can induce the…

Reinforcement Learning with Backtracking Feedback

Evaluated models: Llama 3.2 1B, Llama 3.2 3B, Llama 3 8B Instruct +2 more

Source: arXiv

Related research categories