Impact

Data privacy Research

Security research involving user or training data privacy

11 matching entries out of 114 in this category

Data privacy research results

11 entries

RAG Knowledgebase Exfiltration

Published Feb 10, 2026Analyzed Feb 21, 2026Source: arXiv

Retrieval-Augmented Generation (RAG) systems are vulnerable to iterative knowledge-extraction attacks designed to reconstruct the underlying private knowledge base. The vulnerability exists due to the decoupled optimization of the retrieval and generation phases. Attackers can craft adversarial queries consisting of two distinct components: an "Information" component (optimized via gradient descent or random sampling to steer embeddings toward specific, diverse regions of the vector space) and…

Benchmarking Knowledge-Extraction Attack and Defense on Retrieval-Augmented Generation

Evaluated models: GPT-4o, Llama 3 8B, Qwen 2.5 7B

Drunk Language Jailbreak

Published Jan 19, 2026Analyzed Mar 8, 2026Source: arXiv

A vulnerability exists in aligned Large Language Models (LLMs) where inducing "drunk language" behavior—simulating the text of an intoxicated human—bypasses safety guardrails and contextual privacy protections. Attackers can exploit this anthropomorphic flaw through inference-time persona prompting or lightweight post-training (causal fine-tuning or reinforcement learning on drunk text corpora). By forcing the model to adopt a stylistic and semantic framework associated with impaired human…

In Vino Veritas and Vulnerabilities: Examining LLM Safety via Drunk Language Inducement

Evaluated models: GPT-3.5, GPT-4, GPT-4o+3 more

GPT-3.5
GPT-4
GPT-4o
Llama 2 7B
Llama 3 8B
Mistral 7B

Automated Prompt Recommender Leak

Published Jan 5, 2026Analyzed Feb 21, 2026Source: arXiv

LLaMA-series models (specifically evaluated on LLaMA-1B and LLaMA-3B) exhibit memorization of structured recommender system training data, specifically the MovieLens-1M dataset. While manual prompting yields inconsistent results, the application of Automatic Prompt Engineering (APE)—which treats prompt design as an optimization problem using iterative refinement—allows for the successful extraction of item-level training data (e.g., movie titles and genres) with exact-match accuracy surpassing…

Exploring Approaches for Detecting Memorization of Recommender System Data in Large Language Models

Evaluated models: Not reported

Leaked Bits Collapse Attack Queries

Published Oct 19, 2025Analyzed Dec 9, 2025Source: arXiv

Large Language Models (LLMs), specifically variants of GPT-4o, DeepSeek-R1, OLMo-2, and Llama-4, are vulnerable to accelerated adaptive adversarial attacks due to excessive information leakage in observable output signals. When these models expose "thinking processes" (Chain-of-Thought traces) or token-level log-probabilities (logits) to the end user, they leak significant mutual information $I(Z;T)$ regarding the model's safety state or hidden instructions. This leakage allows adaptive attack…

Bits Leaked per Query: Information-Theoretic Bounds on Adversarial Attacks against LLMs

Evaluated models: DeepSeek R1, GPT-4o Mini 2024-07-18, Llama 4 Maverick 17B+4 more

DeepSeek R1
GPT-4o Mini 2024-07-18
Llama 4 Maverick 17B
Llama 4 Scout 17B
OLMo 2 7B-1124
OLMo 2 13B-1124
OLMo 2 32B-0325

Prompt Injection Alignment Bypass

Published Sep 4, 2025Analyzed Dec 8, 2025Source: arXiv

Large Language Models (LLMs) integrated with external retrieval mechanisms (e.g., Retrieval-Augmented Generation (RAG), web search, or email processing) are vulnerable to Indirect Prompt Injection. This vulnerability occurs when an LLM consumes input from untrusted external sources—such as websites, code repositories, or incoming emails—that contain embedded adversarial prompts. Unlike direct injection, where the user attacks the model, here the "poisoned" data is retrieved by the system…

Breaking to Build: A Threat Model of Prompt-Based Attacks for Securing LLMs

Evaluated models: Not reported

Gradient-Based Privacy Jailbreak

Published May 15, 2025Analyzed May 31, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to a novel privacy jailbreak attack, dubbed PIG (Privacy Jailbreak Attack on LLMs via Gradient-based Iterative In-Context Optimization). PIG leverages in-context learning and gradient-based iterative optimization to extract Personally Identifiable Information (PII) from LLMs, bypassing built-in safety mechanisms. The attack iteratively refines a crafted prompt based on gradient information, focusing on tokens related to PII entities, thereby…

PIG: Privacy Jailbreak Attack on LLMs via Gradient-based Iterative In-Context Optimization

Evaluated models: Claude 3.5 Sonnet, GPT-4o, Llama 2 7B Chat+3 more

Claude 3.5 Sonnet
GPT-4o
Llama 2 7B Chat
Llama 3 8B Instruct
Mistral 7B Instruct v0.3
Vicuna 7B v1.5

LLM Censorship Vector Control

Published Apr 23, 2025Analyzed May 4, 2025Source: arXiv

Large Language Models (LLMs) employing safety mechanisms based on supervised fine-tuning and preference alignment exhibit a vulnerability to "steering" attacks. Maliciously crafted prompts or input manipulations can exploit representation vectors within the model to either bypass censorship ("refusal-compliance vector") or suppress the model's reasoning process ("thought suppression vector"), resulting in the generation of unintended or harmful outputs. This vulnerability is demonstrated…

Steering the CensorShip: Uncovering Representation Vectors for LLM" Thought" Control

Evaluated models: DeepSeek R1 Distill Qwen 1.5B, DeepSeek R1 Distill Qwen 32B, DeepSeek R1 Distill Qwen 7B+8 more

DeepSeek R1 Distill Qwen 1.5B
DeepSeek R1 Distill Qwen 32B
DeepSeek R1 Distill Qwen 7B
Gemma 2B
Gemma 7B
Llama 2 7B
Llama 3.1 8B
Qwen 1.8B
Qwen 2.5 7B
Qwen 7B
Yi 1.5 6B

Cyber Fine-Tuning Undermines Safety

Published Mar 12, 2025Analyzed Jan 14, 2026Source: arXiv

Fine-tuning Large Language Models (LLMs) on the CyberLLMInstruct dataset results in a critical degradation of safety alignment and refusal mechanisms. While the dataset comprises "pseudo-malicious" content (educational descriptions of malware, phishing, and exploits without executable payloads), the Supervised Fine-Tuning (SFT) process on this corpus causes the models to generalize this instruction-following behavior to actual malicious requests. This effectively bypasses safety guardrails…

CyberLLMInstruct: A new dataset for analysing safety of fine-tuned LLMs using cyber security data

Evaluated models: Llama 2 70B, Llama 3 8B, Llama 3.1 8B+4 more

Llama 2 70B
Llama 3 8B
Llama 3.1 8B
Phi-3 Mini 3.8B
Mistral 7B
Qwen 2.5 7B
Gemma 2 9B

Unlearning Relearning Attack

Published Feb 7, 2025Analyzed Dec 30, 2025Source: arXiv

Standard Large Language Model (LLM) unlearning techniques, specifically Negative Preference Optimization (NPO), Gradient Difference (GradDiff), and Representation Misdirection for Unlearning (RMU), fail to sufficiently flatten the loss landscape surrounding the "forgotten" weights. This sharp loss landscape allows for a "Relearning Attack," wherein an attacker can fully restore the unlearned capabilities (such as hazardous knowledge, sensitive data, or copyrighted material) by performing…

Towards llm unlearning resilient to relearning attacks: A sharpness-aware minimization perspective and beyond

Evaluated models: Llama 2 7B, Llama 3 8B

Adversarial Unlearning Bypass

Published Aug 20, 2024Analyzed Jan 26, 2025Source: arXiv

Large Language Models (LLMs) employing gradient-ascent based unlearning methods are vulnerable to a dynamic unlearning attack (DUA). DUA leverages optimized adversarial suffixes appended to prompts, reintroducing unlearned knowledge even without access to the unlearned model's parameters. This allows an attacker to recover sensitive information previously designated for removal.

Towards robust knowledge unlearning: An adversarial framework for assessing and improving unlearning robustness in large language models

Evaluated models: Llama 2 7B Chat, Llama 3 8B Instruct, Llama 3.1 8B Instruct

Related research categories