Attack Type

Extraction Research

Techniques for extracting sensitive information from models

10 matching entries out of 101 in this category

Extraction research results

10 entries

LLM Private Attribute Inference

Published Feb 12, 2026Analyzed Mar 8, 2026Source: arXiv

Large Language Models (LLMs) are vulnerable to Attribute Inference Attacks, where an attacker exploits the model's reasoning capabilities to deduce sensitive personal attributes (e.g., age, gender, location, income level) from seemingly innocuous, unclassified user-generated text. Unlike traditional privacy leaks that rely on the memorization of training data, this vulnerability leverages the model's zero-shot inference and contextual deduction. Because the attack prompts are benign in nature…

Stop Tracking Me! Proactive Defense Against Attribute Inference Attack in LLMs

Evaluated models: Llama 2 7B Chat, Llama 2 13B Chat, Llama 3.1 8B Instruct+5 more

Llama 2 7B Chat
Llama 2 13B Chat
Llama 3.1 8B Instruct
DeepSeek R1 Distill Qwen 7B
Qwen 2.5 7B Instruct
GPT-3.5 Turbo
GPT-4o
Gemini 2.5 Pro

RAG Knowledgebase Exfiltration

Published Feb 10, 2026Analyzed Feb 21, 2026Source: arXiv

Retrieval-Augmented Generation (RAG) systems are vulnerable to iterative knowledge-extraction attacks designed to reconstruct the underlying private knowledge base. The vulnerability exists due to the decoupled optimization of the retrieval and generation phases. Attackers can craft adversarial queries consisting of two distinct components: an "Information" component (optimized via gradient descent or random sampling to steer embeddings toward specific, diverse regions of the vector space) and…

Benchmarking Knowledge-Extraction Attack and Defense on Retrieval-Augmented Generation

Evaluated models: GPT-4o, Llama 3 8B, Qwen 2.5 7B

Leaked Bits Collapse Attack Queries

Published Oct 19, 2025Analyzed Dec 9, 2025Source: arXiv

Large Language Models (LLMs), specifically variants of GPT-4o, DeepSeek-R1, OLMo-2, and Llama-4, are vulnerable to accelerated adaptive adversarial attacks due to excessive information leakage in observable output signals. When these models expose "thinking processes" (Chain-of-Thought traces) or token-level log-probabilities (logits) to the end user, they leak significant mutual information $I(Z;T)$ regarding the model's safety state or hidden instructions. This leakage allows adaptive attack…

Bits Leaked per Query: Information-Theoretic Bounds on Adversarial Attacks against LLMs

Evaluated models: DeepSeek R1, GPT-4o Mini 2024-07-18, Llama 4 Maverick 17B+4 more

DeepSeek R1
GPT-4o Mini 2024-07-18
Llama 4 Maverick 17B
Llama 4 Scout 17B
OLMo 2 7B-1124
OLMo 2 13B-1124
OLMo 2 32B-0325

Prompt Injection Alignment Bypass

Published Sep 4, 2025Analyzed Dec 8, 2025Source: arXiv

Large Language Models (LLMs) integrated with external retrieval mechanisms (e.g., Retrieval-Augmented Generation (RAG), web search, or email processing) are vulnerable to Indirect Prompt Injection. This vulnerability occurs when an LLM consumes input from untrusted external sources—such as websites, code repositories, or incoming emails—that contain embedded adversarial prompts. Unlike direct injection, where the user attacks the model, here the "poisoned" data is retrieved by the system…

Breaking to Build: A Threat Model of Prompt-Based Attacks for Securing LLMs

Evaluated models: Not reported

Obfuscated Activations Jailbreak

Published Dec 12, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to attacks that generate obfuscated activations, bypassing latent-space defenses such as sparse autoencoders, representation probing, and latent out-of-distribution (OOD) detection. Attackers can manipulate model inputs or training data to produce outputs exhibiting malicious behavior while remaining undetected by these defenses. This occurs because the models can represent harmful behavior through diverse activation patterns, allowing attackers to…

Obfuscated Activations Bypass LLM Latent-Space Defenses

Evaluated models: Gemma 2 2B, Llama 3 8B Instruct

Attention-Guided Jailbreak

Published Oct 11, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to jailbreaking attacks that manipulate attention scores to redirect the model's focus away from safety protocols. The AttnGCG attack method increases the attention score on adversarial suffixes within the input prompt, causing the model to prioritize the malicious content over safety guidelines, leading to the generation of harmful outputs.

AttnGCG: Enhancing jailbreaking attacks on LLMs with attention manipulation

Evaluated models: Gemini 1.5 Flash, Gemini Pro, Gemini 1.5 Pro Latest+6 more

Gemini 1.5 Flash
Gemini Pro
Gemini 1.5 Pro Latest
GPT-3.5 Turbo
GPT-4
Llama 2 7B Chat
Mixtral 8x7B Instruct
Vicuna 13B
Vicuna 7B

GCG Suffix Data Exfiltration

Published Aug 1, 2024Analyzed Mar 24, 2025Source: arXiv

A Cross-Prompt Injection Attack (XPIA) can be amplified by appending a Greedy Coordinate Gradient (GCG) suffix to the malicious injection. This increases the likelihood that a Large Language Model (LLM) will execute the injected instruction, even in the presence of a user's primary instruction, leading to data exfiltration. The success rate of the attack depends on the LLM's complexity; medium-complexity models show increased vulnerability.

WHITE PAPER: A Brief Exploration of Data Exfiltration using GCG Suffixes

Evaluated models: GPT-3.5 Turbo, GPT-4o, Phi 3 Mini

Amplified Adversarial Suffix Generation

Published Apr 11, 2024Analyzed Dec 29, 2024Source: arXiv

Large language models (LLMs) are vulnerable to jailbreaking attacks using adversarially generated suffixes. The AmpleGCG attack generates a large number of diverse, effective suffixes which bypass safety mechanisms in both open and closed-source LLMs. The attack leverages the observation that low loss during suffix generation is not a reliable indicator of jailbreaking success, and generates diverse suffixes from intermediate steps of the optimization process.

Amplegcg: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed llms

Evaluated models: GPT-3.5 Turbo, GPT-4, Llama 2 7B Chat+2 more

GPT-3.5 Turbo
GPT-4
Llama 2 7B Chat
Mistral 7B
Vicuna 7B

LLM Causal Neuron Attack

Published Dec 13, 2023Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) such as Llama 2 and Vicuna exhibit a vulnerability where specific layers (e.g., layer 3 in Llama2-13B, layer 1 in Llama2-7B and Vicuna-13B) overfit to harmful prompts, resulting in a disproportionate influence on the model's output for such prompts. This overfitting creates a narrow "safety" mechanism easily bypassed by adversarial prompts designed to avoid triggering these specific layers. Additionally, a single neuron (e.g., neuron 2100 in Llama2 and Vicuna)…

Causality analysis for evaluating the security of large language models

Evaluated models: GPT-3.5 Turbo, Llama 2-13B-chat-hf, Llama 2-7B-chat-hf+1 more

GPT-3.5 Turbo
Llama 2-13B-chat-hf
Llama 2-7B-chat-hf
Vicuna-13B Version 1.5

AutoDAN: Interpretable LLM Jailbreak

Published Oct 23, 2023Analyzed Dec 29, 2024Source: arXiv

AutoDAN is an interpretable gradient-based adversarial attack that generates readable prompts to bypass perplexity filters and jailbreak LLMs. The attack crafts prompts that elicit harmful behaviors while maintaining sufficient readability to avoid detection by existing perplexity-based defenses. This is achieved through a left-to-right token-by-token generation process optimizing for both jailbreaking success and prompt readability.

Autodan: Automatic and interpretable adversarial attacks on large language models

Evaluated models: GPT-3.5 Turbo, GPT-4, Guanaco 7B+4 more

GPT-3.5 Turbo
GPT-4
Guanaco 7B
Llama 2 Chat
Pythia 12B
Vicuna 13B
Vicuna 7B

Related research categories