Architecture Components

Fine-tuning Research

Security research involving model fine-tuning processes

126 matching entries

Fine-tuning research results

126 entries

RL-Tuned LLM Jailbreak

Published 11/1/2024

Analyzed 12/29/2024

LLMStinger uses a reinforcement-learning loop to fine-tune an attacker model that generates adversarial suffixes for jailbreak prompts. Because the approach does not require white-box access to the target, it can adapt existing attacks for both open- and closed-source assistants and bypass otherwise effective refusal behavior.

LLMStinger: Jailbreaking LLMs using RL fine-tuned LLMs

Evaluated models: Claude 2, Gemma 2B IT, GPT-3.5 Turbo +3 more

Source: arXiv

Fine-tuning Safety Bypass

Published 10/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs), even those initially aligned for safety, are vulnerable to having their safety mechanisms compromised through fine-tuning on a small number of adversarially-crafted or even seemingly benign sentences. Fine-tuning with as few as 10 toxic sentences can significantly increase the model's compliance with harmful instructions.

Locking down the finetuned llms safety

Evaluated models: GPT-3.5 Turbo, GPT-4, Llama 3 70B Chat +4 more

Source: arXiv

Fuzzy Backdoor Jailbreak

Published 10/1/2024

Analyzed 12/29/2024

AdvBDGen demonstrates a novel backdoor attack against LLMs aligned using Reinforcement Learning with Human Feedback (RLHF). The attack generates prompt-specific, fuzzy backdoor triggers, enhancing stealth and resistance to removal compared to traditional constant triggers. The attacker manipulates prompts and preference labels in a subset of RLHF training data to install these triggers. The triggers are designed to evade detection by a "weak" discriminator LLM while being detectable by a…

AdvBDGen: Adversarially Fortified Prompt-Specific Fuzzy Backdoor Generator Against LLM Alignment

Evaluated models: BERT, Gemma 7B, GPT-4 +4 more

Source: arXiv

Preference Poisoning Extrapolation

Published 10/1/2024

Analyzed 2/21/2026

Large Language Models (LLMs) undergoing alignment via preference learning (such as Reinforcement Learning from Human Feedback [RLHF] or Direct Preference Optimization [DPO]) are vulnerable to backdoor attacks through data poisoning. An attacker can inject a small percentage (e.g., 3% to 5%) of poisoned data into the preference dataset $\mathcal{D} = \{(x, y_w, y_l)\}$. The attack embeds a specific trigger string into the user query $x$.

Poisonbench: Assessing large language model vulnerability to data poisoning

Evaluated models: Llama 2 7B, Llama 3 8B, Mistral 7B +4 more

Source: arXiv

Fine-Tuning Overrides Safety

Published 9/1/2024

Analyzed 2/2/2025

Fine-tuning an open-source Large Language Model (LLM) such as Llama 3.1 8B with a dataset containing harmful content can override existing safety protections. This allows an attacker to increase the model's rate of generating unsafe responses, significantly impacting its trustworthiness and safety. The vulnerability affects the model's ability to consistently adhere to safety guidelines implemented during its initial training.

Overriding Safety protections of Open-source Models

Evaluated models: Llama 3.1 8B

Source: arXiv

Attack LLMs with Toxic Answers

Published 8/1/2024

Analyzed 8/16/2025

Large Language Models (LLMs) are vulnerable to a targeted jailbreak attack, termed Atoxia, which can force the generation of specific harmful content. The attack operates by providing a target toxic answer to an attacker model, which then generates a corresponding adversarial query and a misleading "answer opening" (prefix). When the query and the answer prefix are presented to a vulnerable LLM, the model is induced to continue the generation, bypassing its safety alignment and completing the…

Atoxia: Red-teaming Large Language Models with Target Toxic Answers

Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4o +5 more

Source: arXiv

LLM Data Poisoning Jailbreak

Published 8/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to a novel attack paradigm, "jailbreak-tuning," which combines data poisoning with jailbreaking techniques to bypass existing safety safeguards. This allows malicious actors to fine-tune LLMs to reliably generate harmful outputs, even when trained on mostly benign data. The vulnerability is amplified in larger LLMs, which are more susceptible to learning harmful behaviors from even minimal exposure to poisoned data.

Data Poisoning in LLMs: Jailbreak-Tuning and Scaling Laws

Evaluated models: GPT-3.5 (GPT-3.5-turbo-0125), GPT-4, GPT-4o +3 more

Source: arXiv

Covert LLM Backdoor Finetuning

Published 6/1/2024

Analyzed 12/29/2024

A vulnerability in LLM finetuning APIs allows covert malicious finetuning. Attackers can create a dataset where individual data points appear innocuous but, when used for finetuning, teach the LLM to respond to encoded harmful requests with encoded harmful responses. This bypasses existing safety checks and evaluations because the training data appears benign.

Covert malicious finetuning: Challenges in safeguarding llm adaptation

Evaluated models: GPT-3.5 Turbo, GPT-4, Llama 2 70B

Source: arXiv

Quantization-Based LLM Poisoning

Published 5/1/2024

Analyzed 12/29/2024

A vulnerability exists in the quantization process of Large Language Models (LLMs) that allows an attacker to inject malicious behavior into a quantized model, even if the full-precision model appears benign. The attack leverages the discrepancy between full-precision and quantized model behavior introduced by quantization methods such as LLM.int8(), NF4, and FP4. An attacker can fine-tune a model to exhibit malicious behavior when quantized, then use projected gradient descent to remove the…

Exploiting LLM Quantization

Evaluated models: Gemma 2B, Phi 3 Mini, Phi-2 +3 more

Source: arXiv

Instruction Tuning Poisoning

Published 2/1/2024

Analyzed 12/28/2024

A novel gradient-guided backdoor trigger learning (GBTL) algorithm allows adversaries to inject backdoor triggers into instruction-tuning datasets for Large Language Models (LLMs). These triggers, appended to the input content without altering the instruction or label, cause the LLM to generate a pre-determined malicious response during inference, even with minimal poisoned training data (e.g., 1%). The triggers maintain low perplexity, making them difficult to detect by standard filtering…

Learning to poison large language models during instruction tuning

Evaluated models: Flan-T5 11B, Flan-T5 3B, Llama 2 13B +1 more

Source: arXiv

Related research categories