Attack Surface

Prompt layer Research

Security research involving prompt handling and processing

610 matching entries

Prompt layer research results

610 entries

Agent Tool Misuse Attacks

Published Oct 19, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Model (LLM) agents are vulnerable to obfuscated adversarial prompts that exploit tool misuse. These prompts, crafted through prompt optimization techniques, force the agent to execute tools (e.g., URL fetching, markdown rendering) in a way that leaks sensitive user data (e.g., PII) without the user's knowledge. The prompts are designed to be visually indistinguishable from benign prompts.

Imprompter: Tricking LLM Agents into Improper Tool Use

Evaluated models: Not reported

Attention-Based LLM Jailbreak

Published Oct 18, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to attention-based jailbreak attacks. Attackers can craft prompts that strategically divert the LLM's attention away from sensitive words, causing the model to overlook malicious intent and generate harmful content. This occurs by leveraging the LLM's attention mechanism to focus on benign parts of the prompt while embedding harmful queries within a seemingly harmless context. The success of the attack is correlated with specific attention…

Feint and Attack: Attention-Based Strategies for Jailbreaking and Protecting LLMs

Evaluated models: Claude 3 Haiku, GPT-4, Llama 2 13B Chat+2 more

Claude 3 Haiku
GPT-4
Llama 2 13B Chat
Llama 2 7B Chat
Llama 3 8B

Robotic LLM Jailbreak

Published Oct 17, 2024Analyzed Dec 28, 2024Source: arXiv

Large language models (LLMs) controlling robots are vulnerable to jailbreaking attacks. The ROBOPAIR algorithm demonstrates that malicious prompts can bypass safety mechanisms, causing robots to perform harmful physical actions. This vulnerability exploits the LLM's reliance on textual prompts and its potential lack of sufficient contextual understanding to prevent unsafe commands. The attack is effective across different access levels.

Jailbreaking LLM-controlled robots

Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4o+1 more

GPT-3.5 Turbo
GPT-4
GPT-4o
Nvidia Dolphins Self-driving Llm

Multi-Turn Question Fragmentation Jailbreak

Published Oct 15, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a multi-turn jailbreak attack, termed "Jigsaw Puzzles" (JSP), which circumvents existing safeguards by splitting harmful questions into harmless fragments. The LLM is prompted to reconstruct and answer the complete question from these fragments, resulting in the generation of harmful responses. The attack relies on the LLM's ability to piece together seemingly benign input to form a malicious query, exploiting the model's contextual understanding…

Jigsaw Puzzles: Splitting Harmful Questions to Jailbreak Large Language Models

Evaluated models: Gemini 1.5 Pro, GPT-4, GPT-4o+2 more

Gemini 1.5 Pro
GPT-4
GPT-4o
GPT-4o Mini
Llama 3.1 70B

Prompt Translation Jailbreak

Published Oct 15, 2024Analyzed Dec 29, 2024Source: arXiv

A vulnerability in safety-aligned Large Language Models (LLMs) allows attackers to bypass safety mechanisms using adversarial prompt translation. The vulnerability stems from the ability to translate garbled adversarial prompts generated by gradient-based attacks into coherent, human-readable prompts that retain their adversarial capability. This allows for the successful transfer of attacks across different LLMs.

Deciphering the Chaos: Enhancing Jailbreak Attacks via Adversarial Prompt Translation

Evaluated models: Not reported

Multi-turn Actor Jailbreak

Published Oct 14, 2024Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to multi-turn adversarial attacks where malicious users obscure harmful intents across multiple queries. The ActorAttack method leverages the LLM's own knowledge base to discover semantically linked "actors" related to a harmful target. By posing seemingly innocuous questions about these actors, the attacker guides the LLM towards revealing harmful information step-by-step, accumulating knowledge until the desired malicious output is obtained, even…

Derail Yourself: Multi-turn LLM Jailbreak Attack through Self-discovered Clues

Evaluated models: Claude 3.5 Sonnet, GPT-3.5 Turbo, GPT-4+3 more

Claude 3.5 Sonnet
GPT-3.5 Turbo
GPT-4
Llama 3 70B
Llama 3 8B
o1

Multi-Objective LLM Jailbreak

Published Oct 13, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a multi-objective black-box jailbreaking attack (BlackDAN) that optimizes prompts to maximize the likelihood of generating unsafe responses while maintaining contextual relevance and minimizing detectability. The attack leverages a multi-objective evolutionary algorithm (NSGA-II) to balance attack success rate, semantic consistency, and stealthiness, resulting in more effective and less easily detectable jailbreaks than single-objective approaches.

BlackDAN: A Black-Box Multi-Objective Approach for Effective and Contextual Jailbreaking of Large Language Models

Evaluated models: Aquilachat-7B, Baichuan 2 13B Chat, Baichuan-7B+12 more

Aquilachat-7B
Baichuan 2 13B Chat
Baichuan-7B
GPT-2 XL
Internlm2-chat-7B
Llama 3 8B
Llama-2-13B-hf
Llama-2-7B-hf
Llava v1.6-mistral-7B-hf
Llava-v1.6-vicuna-7B-hf
Minitron-8B-base
Vicuna 13B v1.5
Vicuna 7B
Vicuna 7B v1.5
Yi 1.5 9B Chat

Context-Shifting Code Injection

Published Oct 9, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) acting as code assistants may recommend malicious code or resources when presented with prompts framed as programming challenges, even if they refuse similar direct prompts. This occurs due to insufficient context-aware safety mechanisms. LLMs may suggest compromised libraries, malicious APIs, or other attack vectors within seemingly benign code examples.

Hallucinating AI Hijacking Attack: Large Language Models and Malicious Code Recommenders

Evaluated models: GPT-4

LLM Resource Exhaustion Jailbreak

Published Oct 5, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a novel jailbreak attack that exploits resource limitations. By overloading the model with a computationally intensive preliminary task (e.g., a complex character map lookup and decoding), the attacker prevents the activation of the LLM's safety mechanisms, enabling the generation of unsafe outputs from subsequent prompts. The attack's strength is scalable and adjustable by modifying the complexity of the preliminary task.

Harnessing Task Overload for Scalable Jailbreak Attacks on Large Language Models

Evaluated models: Llama 3 8B, Mistral 7B, Qwen 2.5 14B+5 more

Llama 3 8B
Mistral 7B
Qwen 2.5 14B
Qwen 2.5 32B
Qwen 2.5 7B
Qwen 2.5 3B
Qwen 2.5 72B
Vicuna7B-v0.3

Implicit Reference Jailbreak

Published Oct 4, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to an attack vector termed "Attack via Implicit Reference" (AIR). AIR bypasses safety mechanisms by decomposing a malicious objective into multiple benign, seemingly unrelated objectives linked through implicit contextual references. The LLM generates harmful content by combining the outputs of these seemingly harmless objectives, without explicitly triggering safety filters designed to detect direct requests for malicious content.

You Know What I'm Saying: Jailbreak Attack via Implicit Reference

Evaluated models: Claude 3.5 Sonnet, GPT-4o, GPT-4o Mini+7 more

Claude 3.5 Sonnet
GPT-4o
GPT-4o Mini
GPT-4o-0513
Llama 3 70B
Llama 3 8B
Qwen 2 7B
Qwen 2 0.5B
Qwen 2 1.5B
Qwen 2 72B

Related research categories