Impact

Data privacy Research

Security research involving user or training data privacy

4 matching entries out of 114 in this category

Data privacy research results

4 entries

RAG Knowledgebase Exfiltration

Published Feb 10, 2026Analyzed Feb 21, 2026Source: arXiv

Retrieval-Augmented Generation (RAG) systems are vulnerable to iterative knowledge-extraction attacks designed to reconstruct the underlying private knowledge base. The vulnerability exists due to the decoupled optimization of the retrieval and generation phases. Attackers can craft adversarial queries consisting of two distinct components: an "Information" component (optimized via gradient descent or random sampling to steer embeddings toward specific, diverse regions of the vector space) and…

Benchmarking Knowledge-Extraction Attack and Defense on Retrieval-Augmented Generation

Evaluated models: GPT-4o, Llama 3 8B, Qwen 2.5 7B

Prompt Injection Alignment Bypass

Published Sep 4, 2025Analyzed Dec 8, 2025Source: arXiv

Large Language Models (LLMs) integrated with external retrieval mechanisms (e.g., Retrieval-Augmented Generation (RAG), web search, or email processing) are vulnerable to Indirect Prompt Injection. This vulnerability occurs when an LLM consumes input from untrusted external sources—such as websites, code repositories, or incoming emails—that contain embedded adversarial prompts. Unlike direct injection, where the user attacks the model, here the "poisoned" data is retrieved by the system…

Breaking to Build: A Threat Model of Prompt-Based Attacks for Securing LLMs

Evaluated models: Not reported

Cyber Fine-Tuning Undermines Safety

Published Mar 12, 2025Analyzed Jan 14, 2026Source: arXiv

Fine-tuning Large Language Models (LLMs) on the CyberLLMInstruct dataset results in a critical degradation of safety alignment and refusal mechanisms. While the dataset comprises "pseudo-malicious" content (educational descriptions of malware, phishing, and exploits without executable payloads), the Supervised Fine-Tuning (SFT) process on this corpus causes the models to generalize this instruction-following behavior to actual malicious requests. This effectively bypasses safety guardrails…

CyberLLMInstruct: A new dataset for analysing safety of fine-tuned LLMs using cyber security data

Evaluated models: Llama 2 70B, Llama 3 8B, Llama 3.1 8B+4 more

Llama 2 70B
Llama 3 8B
Llama 3.1 8B
Phi-3 Mini 3.8B
Mistral 7B
Qwen 2.5 7B
Gemma 2 9B

GCG Suffix Data Exfiltration

Published Aug 1, 2024Analyzed Mar 24, 2025Source: arXiv

A Cross-Prompt Injection Attack (XPIA) can be amplified by appending a Greedy Coordinate Gradient (GCG) suffix to the malicious injection. This increases the likelihood that a Large Language Model (LLM) will execute the injected instruction, even in the presence of a user's primary instruction, leading to data exfiltration. The success rate of the attack depends on the LLM's complexity; medium-complexity models show increased vulnerability.

WHITE PAPER: A Brief Exploration of Data Exfiltration using GCG Suffixes

Evaluated models: GPT-3.5 Turbo, GPT-4o, Phi 3 Mini

Related research categories