Attack Surface

Prompt layer Research

Security research involving prompt handling and processing

610 matching entries

Prompt layer research results

610 entries

Cross-Batch Interference

Published Mar 18, 2025Analyzed Dec 30, 2025Source: arXiv

Large Language Models (LLMs) deployed using "Batch Prompting" strategies—where multiple distinct user queries are concatenated and processed in a single inference pass to reduce computational costs—are vulnerable to Cross-Query Prompt Injection. When a batch contains a mixture of benign queries and a single malicious query, the instructions within the malicious query (e.g., "apply this rule to every answer") bleed over the context window. This causes the model to apply the adversary's…

Efficient but Vulnerable: Benchmarking and Defending LLM Batch Prompting Attack

Evaluated models: GPT-4o, GPT-4o Mini, Claude 3.5 Sonnet+4 more

GPT-4o
GPT-4o Mini
Claude 3.5 Sonnet
Llama 3 70B Instruct
Llama 3.2 3B Instruct
Qwen 2.5 7B Instruct
DeepSeek R1

Agent System Orchestration Hijack

Published Mar 15, 2025Analyzed Jan 14, 2026Source: arXiv

Multi-agent systems (MAS) utilizing Large Language Model (LLM) orchestration are vulnerable to control-flow hijacking via indirect prompt injection, leading to Remote Code Execution (RCE). This vulnerability arises when a sub-agent (e.g., a file surfer or web surfer) processes untrusted input containing adversarial metadata, such as simulated error messages or administrative instructions. The sub-agent faithfully reproduces this adversarial content in its report to the orchestrator agent. The…

Multi-agent systems execute arbitrary malicious code

Evaluated models: GPT-4o, GPT-4o Mini, Gemini 1.5 Pro+1 more

GPT-4o
GPT-4o Mini
Gemini 1.5 Pro
Gemini 1.5 Flash

Autonomous Multi-Turn LLM Jailbreak

Published Mar 13, 2025Analyzed Mar 19, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to multi-turn adversarial attacks that exploit incremental policy erosion. The attacker uses a breadth-first search strategy to generate multiple prompts at each turn, leveraging partial compliance from previous responses to gradually escalate the conversation towards eliciting disallowed outputs. Minor concessions accumulate, ultimately leading to complete circumvention of safety measures.

Siege: Autonomous Multi-Turn Jailbreaking of Large Language Models with Tree Search

Evaluated models: GPT-3.5 Turbo, GPT-4, Llama 3.1 70B

Cyber Fine-Tuning Undermines Safety

Published Mar 12, 2025Analyzed Jan 14, 2026Source: arXiv

Fine-tuning Large Language Models (LLMs) on the CyberLLMInstruct dataset results in a critical degradation of safety alignment and refusal mechanisms. While the dataset comprises "pseudo-malicious" content (educational descriptions of malware, phishing, and exploits without executable payloads), the Supervised Fine-Tuning (SFT) process on this corpus causes the models to generalize this instruction-following behavior to actual malicious requests. This effectively bypasses safety guardrails…

CyberLLMInstruct: A new dataset for analysing safety of fine-tuned LLMs using cyber security data

Evaluated models: Llama 2 70B, Llama 3 8B, Llama 3.1 8B+4 more

Llama 2 70B
Llama 3 8B
Llama 3.1 8B
Phi-3 Mini 3.8B
Mistral 7B
Qwen 2.5 7B
Gemma 2 9B

LLM Fuzz-Based Jailbreak

Published Mar 12, 2025Analyzed Mar 19, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to jailbreak attacks by crafted prompts that bypass safety mechanisms, causing the model to generate harmful or unethical content. This vulnerability stems from the inherent tension between the LLM's instruction-following and safety constraints. The JBFuzz technique demonstrates the ability to efficiently and effectively discover such prompts through a fuzzing-based approach leveraging novel seed prompt templates and a synonym-based mutation strategy.

JBFuzz: Jailbreaking LLMs Efficiently and Effectively Using Fuzzing

Evaluated models: DeepSeek Chat, DeepSeek R1, Gemini 1.5 Flash+6 more

DeepSeek Chat
DeepSeek R1
Gemini 1.5 Flash
Gemini 2.0 Flash
GPT-3.5 Turbo
GPT-4o
GPT-4o Mini
Llama 2 7B Chat
Llama 3.1 8B Instruct

Unchallenged Premise Misinformation

Published Mar 12, 2025Analyzed Dec 9, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to implicit misinformation propagation due to sycophantic compliance with false premises. When a user prompt embeds a factually incorrect assumption or conspiracy theory as an unchallenged premise (implicit presupposition) rather than asking for verification, the model frequently fails to detect the falsehood. Instead of correcting the user, the model hallucinates a response that accepts, validates, and reinforces the false premise. This…

How to Protect Yourself from 5G Radiation? Investigating LLM Responses to Implicit Misinformation

Evaluated models: Gemini 1.5 Pro, Gemini 2.0 Flash, Claude 3.5 Sonnet+11 more

Gemini 1.5 Pro
Gemini 2.0 Flash
Claude 3.5 Sonnet
GPT-4
GPT-4o
o1
Mixtral 8x7B
Qwen 2.5 7B
Qwen 2.5 72B
Tülu 3 8B
Tülu 3 70B
Llama 3.1 8B
Llama 3.1 70B
Llama 3.3 70B

Dialogue History Jailbreak

Published Mar 11, 2025Analyzed Mar 19, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to Dialogue Injection Attacks (DIA), where malicious actors manipulate the chat history to bypass safety mechanisms and elicit harmful or unethical responses. DIA exploits the LLM's chat template structure to inject crafted dialogue into the input, even in black-box scenarios where the model's internals are unknown. Two attack methods are presented: one adapts gray-box prefilling attacks, the other leverages deferred responses to increase the…

Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulation

Evaluated models: Gemma 2 27B, Gemma 2 2B, Gemma 2 9B+8 more

Gemma 2 27B
Gemma 2 2B
Gemma 2 9B
GPT-4o
GPT-4o Mini
Llama 2 7B
Llama 3 70B
Llama 3 8B
Llama 3.1 8B
Llama 3.2 11B
Qwen 2 7B

Life-Cycle Router Misrouting

Published Mar 9, 2025Analyzed Dec 30, 2025Source: arXiv

Predictive Large Language Model (LLM) routers, specifically those utilizing Deep Neural Network (DNN) and Matrix Factorization (MF) architectures, are vulnerable to adversarial manipulation and backdoor poisoning. These routers are designed to optimize cost and latency by dynamically directing simple queries to "weak" (cheap) models and complex queries to "strong" (expensive) models. Attackers can exploit this mechanism in two ways: 1. Inference-time Attacks: By appending specific adversarial…

Life-Cycle Routing Vulnerabilities of LLM Router

Evaluated models: Not reported

Cross-Modal Toxic Continuation

Published Mar 8, 2025Analyzed Dec 9, 2025Source: arXiv

Large Vision-Language Models (VLMs) are vulnerable to a cross-modal toxic continuation attack facilitated by reinforcement learning-tuned diffusion models. This vulnerability allows an attacker to bypass safety alignment and external guardrails (such as NSFW image filters) by pairing a specific text prefix with a "semantically adversarial" image. Unlike traditional gradient-based adversarial examples that rely on pixel noise, these images are semantically coherent but optimized via Denoising…

RedDiffuser: Auditing Multimodal Safety Failures in Vision-Language Models via Reinforced Diffusion

Evaluated models: LLaVA 1.5 7B, Gemini 1.5 Flash, Llama 3.2 11B Vision Instruct

Image-Text Jailbreak

Published Mar 5, 2025Analyzed Mar 8, 2026Source: arXiv

Multimodal Large Language Models (MLLMs) are vulnerable to coupled cross-modal jailbreak attacks that combine continuous visual perturbations with discrete textual manipulations. Because standard alignment and single-modality defenses (such as text-only safety tuning or isolated vision-encoder adversarial training) fail to secure the cross-modal interaction, attackers can simultaneously apply gradient-based noise (e.g., PGD) to input images and adversarial suffixes (e.g., GCG) to text prompts…

E2AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Models

Evaluated models: LLaVA 1.5 7B, Bunny 1.0 4B, Mplug-owl2

Related research categories