Attack Type

Poisoning Research

Attacks targeting training data, fine-tuning, retrieval knowledge bases, or persistent agent memory

114 matching entries

Poisoning research results

114 entries

Fine-tuning Compromises LLM Safety

Published Oct 5, 2023Analyzed Dec 28, 2024Source: arXiv

Fine-tuning aligned Large Language Models (LLMs) on a small number of adversarially crafted examples, or even on benign datasets, can compromise their safety alignment, leading to the generation of harmful or inappropriate content. This vulnerability exploits the few-shot learning capabilities of LLMs, allowing attackers to override existing safety mechanisms with minimal effort and cost. Even unintentional fine-tuning with seemingly benign datasets can result in unintended safety degradation.

Fine-tuning aligned language models compromises safety, even when users do not intend to!

Evaluated models: GPT-3.5 Turbo, Llama 2 13B Chat, Llama 2 70B Chat+2 more

GPT-3.5 Turbo
Llama 2 13B Chat
Llama 2 70B Chat
Llama 2 7B Chat
Llama-1

LLM Causal Neuron Attack

Published Dec 13, 2023Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) such as Llama 2 and Vicuna exhibit a vulnerability where specific layers (e.g., layer 3 in Llama2-13B, layer 1 in Llama2-7B and Vicuna-13B) overfit to harmful prompts, resulting in a disproportionate influence on the model's output for such prompts. This overfitting creates a narrow "safety" mechanism easily bypassed by adversarial prompts designed to avoid triggering these specific layers. Additionally, a single neuron (e.g., neuron 2100 in Llama2 and Vicuna)…

Causality analysis for evaluating the security of large language models

Evaluated models: GPT-3.5 Turbo, Llama 2-13B-chat-hf, Llama 2-7B-chat-hf+1 more

GPT-3.5 Turbo
Llama 2-13B-chat-hf
Llama 2-7B-chat-hf
Vicuna-13B Version 1.5

Parametric Model Unalignment

Published Oct 22, 2023Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) employing Reinforcement Learning from Human Feedback (RLHF) and instruction tuning methods may exhibit superficial safety guardrails vulnerable to parametric red-teaming attacks. Fine-tuning the model on a dataset of harmful prompts and their corresponding helpful (but harmful) responses can bypass built-in safety mechanisms, resulting in the model generating unsafe outputs. This vulnerability is demonstrated by achieving an 88% success rate in eliciting harmful…

Language model unalignment: Parametric red-teaming to expose hidden harms and biases

Evaluated models: Claude 1, Claude 2, GPT-4+6 more

Claude 1
Claude 2
GPT-4
Llama 2 7B Chat
Llama 2 13B Chat
Vicuna 1 7B
Vicuna 1 13B
Vicuna 2 7B
Vicuna 2 13B

LLM Memory Poisoning Attack

Published Jul 17, 2024Analyzed Dec 28, 2024Source: arXiv

A vulnerability in Retrieval-Augmented Generation (RAG)-based Large Language Model (LLM) agents allows attackers to inject malicious demonstrations into the agent's memory or knowledge base. By crafting a carefully optimized trigger, an attacker can manipulate the agent's retrieval mechanism to preferentially retrieve these poisoned demonstrations, causing the agent to produce adversarial outputs or take malicious actions even when seemingly benign prompts are used. The attack, termed…

Agentpoison: Red-teaming llm agents via poisoning memory or knowledge bases

Evaluated models: GPT-2, GPT-3.5 Turbo, Llama 3 70B+2 more

GPT-2
GPT-3.5 Turbo
Llama 3 70B
Llama 3 8B
text-embedding-ada-002

Related research categories