Impact

Data security Research

Issues affecting data confidentiality and integrity

171 matching entries

Data security research results

171 entries

Agent Red-Teaming via Fuzzing

Published 5/1/2025

Analyzed 7/14/2025

Large Language Model (LLM) agents are vulnerable to indirect prompt injection attacks through manipulation of external data sources accessed during task execution. Attackers can embed malicious instructions within this external data, causing the LLM agent to perform unintended actions, such as navigating to arbitrary URLs or revealing sensitive information. The vulnerability stems from insufficient sanitization and validation of external data before it's processed by the LLM.

AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents

Evaluated models: Claude 3.5 Sonnet, Gemini 2.0 Flash, GPT-4o +3 more

Source: arXiv

DNA Model Pathogen Synthesis

Published 5/1/2025

Analyzed 6/11/2025

DNA language models, such as the Evo series, are vulnerable to jailbreak attacks that coerce the generation of DNA sequences with high homology to known human pathogens. The GeneBreaker framework demonstrates this by using a combination of carefully crafted prompts leveraging high-homology non-pathogenic sequences and a beam search guided by pathogenicity prediction models (e.g., PathoLM) and log-probability heuristics. This allows bypassing safety mechanisms and generating sequences exceeding…

GeneBreaker: Jailbreak Attacks against DNA Language Models with Pathogenicity Guidance

Evaluated models: Evo1 7B, Evo2 1B, Evo2 7B +2 more

Source: arXiv

Gradient-Based Privacy Jailbreak

Published 5/1/2025

Analyzed 5/31/2025

Large Language Models (LLMs) are vulnerable to a novel privacy jailbreak attack, dubbed PIG (Privacy Jailbreak Attack on LLMs via Gradient-based Iterative In-Context Optimization). PIG leverages in-context learning and gradient-based iterative optimization to extract Personally Identifiable Information (PII) from LLMs, bypassing built-in safety mechanisms. The attack iteratively refines a crafted prompt based on gradient information, focusing on tokens related to PII entities, thereby…

PIG: Privacy Jailbreak Attack on LLMs via Gradient-based Iterative In-Context Optimization

Evaluated models: Claude 3.5 Sonnet, GPT-4o, Llama 2 7B Chat +3 more

Source: arXiv

Informed Adversary LLM Jailbreak

Published 5/1/2025

Analyzed 5/31/2025

Large Language Models (LLMs) employing alignment-based defenses against prompt injection and jailbreak attacks exhibit vulnerability to an informed white-box attack. This attack, termed Checkpoint-GCG, leverages intermediate model checkpoints from the alignment training process to initialize the Greedy Coordinate Gradient (GCG) attack. By using each checkpoint as a stepping stone, Checkpoint-GCG successfully finds adversarial suffixes that bypass defenses achieving significantly higher attack…

Alignment Under Pressure: The Case for Informed Adversaries When Evaluating LLM Defenses

Evaluated models: GPT-3.5 Turbo, GPT-4o, Llama 3 8B Instruct +1 more

Source: arXiv

LLM Multi-Agent IP Leakage

Published 5/1/2025

Analyzed 5/31/2025

Large Language Model (LLM)-based Multi-Agent Systems (MAS) are vulnerable to intellectual property (IP) leakage attacks. An attacker with black-box access (only interacting via the public API) can craft adversarial queries that propagate through the MAS, extracting sensitive information such as system prompts, task instructions, tool specifications, number of agents, and system topology.

IP Leakage Attacks Targeting LLM-Based Multi-Agent Systems

Evaluated models: GPT-4o, GPT-4o Mini, Llama 3.1 70B +2 more

Source: arXiv

LLM System Prompt Extraction

Published 5/1/2025

Analyzed 12/30/2025

Large Language Models (LLMs), including Llama-3, Falcon-3, Gemma-2, and GPT-4 variants, are susceptible to system prompt extraction attacks. The vulnerability exists due to the models' instruction-following nature, which allows remote attackers to bypass safety guardrails and retrieve the model's hidden system configuration (system prompt) verbatim. This is successfully exploited using an "Extended Sandwich Attack," where an adversarial extraction command is embedded between benign questions…

System Prompt Extraction Attacks and Defenses in Large Language Models

Evaluated models: GPT-4, GPT-4o, Llama 3 8B +2 more

Source: arXiv

LLM Censorship Vector Control

Published 4/1/2025

Analyzed 5/4/2025

Large Language Models (LLMs) employing safety mechanisms based on supervised fine-tuning and preference alignment exhibit a vulnerability to "steering" attacks. Maliciously crafted prompts or input manipulations can exploit representation vectors within the model to either bypass censorship ("refusal-compliance vector") or suppress the model's reasoning process ("thought suppression vector"), resulting in the generation of unintended or harmful outputs. This vulnerability is demonstrated…

Steering the CensorShip: Uncovering Representation Vectors for LLM" Thought" Control

Evaluated models: DeepSeek R1 Distill Qwen 1.5B, DeepSeek R1 Distill Qwen 32B, DeepSeek R1 Distill Qwen 7B +8 more

Source: arXiv

Agent Reasoning Hijacking

Published 3/1/2025

Analyzed 3/19/2025

A vulnerability exists in Large Language Model (LLM) agents that allows attackers to manipulate the agent's reasoning process through the insertion of strategically placed adversarial strings. This allows attackers to induce the agent to perform unintended malicious actions or invoke specific malicious tools, even when the initial prompt or instruction is benign. The attack exploits the agent's reliance on chain-of-thought reasoning and dynamically optimizes the adversarial string to maximize…

UDora: A Unified Red Teaming Framework against LLM Agents by Dynamically Hijacking Their Own Reasoning

Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4o +2 more

Source: arXiv

Agent System Orchestration Hijack

Published 3/1/2025

Analyzed 1/14/2026

Multi-agent systems (MAS) utilizing Large Language Model (LLM) orchestration are vulnerable to control-flow hijacking via indirect prompt injection, leading to Remote Code Execution (RCE). This vulnerability arises when a sub-agent (e.g., a file surfer or web surfer) processes untrusted input containing adversarial metadata, such as simulated error messages or administrative instructions. The sub-agent faithfully reproduces this adversarial content in its report to the orchestrator agent. The…

Multi-agent systems execute arbitrary malicious code

Evaluated models: GPT-4o, GPT-4o Mini, Gemini 1.5 Pro +1 more

Source: arXiv

Agent Pipeline Simple Hacks

Published 2/1/2025

Analyzed 12/9/2025

Commercial LLM-powered agents utilizing autonomous web access, memory modules, and retrieval-augmented generation (RAG) are vulnerable to indirect prompt injection and environmental manipulation. Attackers can embed malicious instructions into external data sources trusted by the agent (such as Reddit posts, public databases, or ArXiv papers). When the agent autonomously retrieves and processes this content during task execution, it executes the embedded malicious commands. This vulnerability…

Commercial llm agents are already vulnerable to simple yet dangerous attacks

Evaluated models: Not reported

Source: arXiv

Related research categories