Architecture Components

Fine-tuning Research

Security research involving model fine-tuning processes

126 matching entries

Fine-tuning research results

126 entries

Low-Resource Language Toxicity

Published Sep 18, 2025Analyzed Sep 30, 2025Source: arXiv

Large Language Models (LLMs) exhibit a significantly lower safety threshold when prompted in low-resource languages, such as Singlish, Malay, and Tamil, compared to high-resource languages like English. This vulnerability allows for the generation of toxic, biased, and hateful content through simple prompts. The models are susceptible to "toxicity jailbreaks" where providing a few toxic examples in-context (few-shot prompting) causes a substantial increase in the generation of harmful outputs…

Toxicity Red-Teaming: Benchmarking LLM Safety in Singapore's Low-Resource Languages

Evaluated models: GPT-4o Mini, Llama 3.1 8B Instruct, Mistral 7B Instruct v0.3+3 more

GPT-4o Mini
Llama 3.1 8B Instruct
Mistral 7B Instruct v0.3
Qwen 2.5 7B Instruct
Sea-Lion v2 Instruct
SeaLLM v3 7B Chat

Prompt Injection Alignment Bypass

Published Sep 4, 2025Analyzed Dec 8, 2025Source: arXiv

Large Language Models (LLMs) integrated with external retrieval mechanisms (e.g., Retrieval-Augmented Generation (RAG), web search, or email processing) are vulnerable to Indirect Prompt Injection. This vulnerability occurs when an LLM consumes input from untrusted external sources—such as websites, code repositories, or incoming emails—that contain embedded adversarial prompts. Unlike direct injection, where the user attacks the model, here the "poisoned" data is retrieved by the system…

Breaking to Build: A Threat Model of Prompt-Based Attacks for Securing LLMs

Evaluated models: Not reported

Knowledge Neuron Jailbreak

Published Sep 1, 2025Analyzed Dec 8, 2025Source: arXiv

Aligned Large Language Models (LLMs) utilizing Transformer architectures are vulnerable to representation-level attacks targeting safety-knowledge neurons within the Multi-Layer Perceptron (MLP) layers. Research indicates that safety decision-making (Rejection vs. Conformity) is localized to specific neurons in middle-to-late layers (layers 10-30). An attacker with white-box access can calculate a "Conformity" direction vector based on the activation differences between benign and harmful…

Unraveling LLM Jailbreaks Through Safety Knowledge Neurons

Evaluated models: Llama 2 7B, Vicuna 7B

Optimization-Induced Safety Degradation

Published Aug 17, 2025Analyzed Dec 30, 2025Source: arXiv

Safety alignment degradation occurs in instruction-tuned Large Language Models (LLMs), specifically Llama-2-7B, Llama-3.2-1B, Qwen2.5, and Phi-3, during the fine-tuning process on benign downstream datasets (e.g., Dolly, Alpaca). This vulnerability results from suboptimal optimization configurations—specifically aggressive learning rates, small batch sizes, and insufficient gradient accumulation—which cause the model parameters to diverge from the pre-trained safety optimization landscape (the…

Rethinking safety in llm fine-tuning: An optimization perspective

Evaluated models: GPT-4, GPT-4o, Llama 2 7B+3 more

GPT-4
GPT-4o
Llama 2 7B
Llama 3.2 1B
Qwen 2.5 3B
Phi-3 3B

Magic Token Safety Bypass

Published Aug 12, 2025Analyzed Feb 21, 2026Source: arXiv

The Magic-Token-Guided Co-Training (MTC) framework for Large Language Models (LLMs) introduces a mechanism where distinct behavioral modes are activated via hardcoded system-level strings known as "magic tokens." A specific vulnerability exists in the implementation of the "negative" (neg) behavior mode, which is explicitly trained to generate unfiltered, risk-prone, and harmful content for internal red-teaming. The framework relies on the secrecy of the magic token (e.g., a random string like…

Efficient Switchable Safety Control in LLMs via Magic-Token-Guided Co-Training

Evaluated models: Qwen 3 8B

Stealthy Multi-Round Communication Tampering

Published Aug 5, 2025Analyzed Aug 16, 2025Source: arXiv

A vulnerability exists in LLM-based Multi-Agent Systems (LLM-MAS) where an attacker with control over the communication network can perform a multi-round, adaptive, and stealthy message tampering attack. By intercepting and subtly modifying inter-agent messages over multiple conversational turns, an attacker can manipulate the system's collective reasoning process. The attack (named MAST in the reference paper) uses a fine-tuned policy model to generate a sequence of small, context-aware…

Attack the Messages, Not the Agents: A Multi-round Adaptive Stealthy Tampering Framework for LLM-MAS

Evaluated models: Gemini 2.5 Pro, GPT-4o, Llama 3.1 70B Instruct+3 more

Gemini 2.5 Pro
GPT-4o
Llama 3.1 70B Instruct
Llama 3.1 8B Instruct
Mistral 7B Instruct v0.3
Qwen 3 8B

Simulated Ensemble Jailbreak Transfer

Published Aug 3, 2025Analyzed Dec 30, 2025Source: arXiv

A vulnerability exists in the fine-tuning lifecycle of Vision-Language Models (VLMs) derived from open-source base models, termed the "grey-box threat." Adversaries with white-box access to a public base model (e.g., Qwen2-VL) can generate universal adversarial images that successfully bypass safety guardrails in proprietary, fine-tuned downstream variants. This is achieved via the Simulated Ensemble Attack (SEA), which combines two techniques: Fine-tuning Trajectory Simulation (FTS), where…

Simulated Ensemble Attack: Transferring Jailbreaks Across Fine-tuned Vision-Language Models

Evaluated models: Qwen 2 2B

Enterprise Multi-Turn Data Exfiltration

Published Jul 21, 2025Analyzed Jul 28, 2025Source: arXiv

Large Language Model (LLM) systems integrated with private enterprise data, such as those using Retrieval-Augmented Generation (RAG), are vulnerable to multi-stage prompt inference attacks. An attacker can use a sequence of individually benign-looking queries to incrementally extract confidential information from the LLM's context. Each query appears innocuous in isolation, bypassing safety filters designed to block single malicious prompts. By chaining these queries, the attacker can…

Multi-Stage Prompt Inference Attacks on Enterprise LLM Systems

Evaluated models: GPT-2, GPT-3, GPT-4+1 more

GPT-2
GPT-3
GPT-4
RoBERTa

Patching Induces Emergent Deception

Published Jul 12, 2025Analyzed Dec 30, 2025Source: arXiv

Adversarial Activation Patching enables the induction of emergent deceptive behaviors in safety-aligned transformer-based Large Language Models (LLMs). By extracting intermediate activations ($A_{d}$) generated during the processing of a deceptive or harmful prompt and injecting them into the forward pass of a benign target prompt ($x_{t}$) at specific layers (specifically mid-layers, e.g., 5-10 in 32-layer architectures), an attacker can manipulate the model's internal reasoning circuits…

Adversarial activation patching: A framework for detecting and mitigating emergent deception in safety-aligned transformers

Evaluated models: GPT-4

Response-Primed LLM Jailbreak

Published Jul 7, 2025Analyzed Aug 31, 2025Source: arXiv

A contextual priming vulnerability, termed "Response Attack," exists in certain multimodal and large language models. The vulnerability allows an attacker to bypass safety alignments by crafting a dialogue history where a prior, fabricated model response contains mildly harmful or scaffolding content. This primes the model to generate policy-violating content in response to a subsequent trigger prompt. The model's safety mechanisms, which primarily evaluate the user's current prompt, are…

Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Models

Evaluated models: DeepSeek R1 Distill Llama 70B, Gemini 2.0 Flash, Gemini 2.5 Flash+5 more

DeepSeek R1 Distill Llama 70B
Gemini 2.0 Flash
Gemini 2.5 Flash
GPT-4.1
GPT-4o
Llama 3 70B Instruct
Llama 3 8B Instruct
QwQ 32B

Related research categories