Attack Surface

Prompt layer Research

Security research involving prompt handling and processing

610 matching entries

Prompt layer research results

610 entries

Sequential Prompt Jailbreak

Published Nov 10, 2024Analyzed Jan 26, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to "SequentialBreak," a jailbreak attack where embedding a harmful prompt within a chain of benign prompts in a single query can bypass LLM safety features. The LLM's attention mechanism prioritizes the benign prompts, allowing the harmful prompt to be processed without triggering safety mitigations.

SequentialBreak: Large Language Models Can be Fooled by Embedding Jailbreak Prompts into Sequential Prompt Chains

Evaluated models: Not reported

Stochastic Monkey Jailbreak

Published Nov 5, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) employing safety alignment mechanisms are vulnerable to a bypass attack using simple, stochastic random augmentations of input prompts. The attack leverages the inherent brittleness of safety alignment to minor, randomly introduced modifications in the input, causing the LLM to generate unsafe outputs despite its safety training. Character-level augmentations prove significantly more effective than string insertions.

Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignment

Evaluated models: GPT-4o, Llama 2 13B Chat, Llama 2 7B Chat+12 more

GPT-4o
Llama 2 13B Chat
Llama 2 7B Chat
Llama 3 8B Instruct
Llama 3.1 8B Instruct
Mistral 7B Instruct v0.2
Phi 3 Medium 4k Instruct
Phi 3 Mini 4k Instruct
Phi 3 Small 8k Instruct
Qwen 2 0.5B
Qwen 2 1.5B
Qwen 2 7B
Vicuna 13B v1.5
Vicuna 7B v1.5
Zephyr 7B Beta

BarkPlug Data Poisoning Attack

Published Nov 3, 2024Analyzed Mar 19, 2025Source: arXiv

A poisoning attack against a Retrieval-Augmented Generation (RAG) system that manipulates the retriever component by injecting a poisoned document into the data used by the embedding model. This poisoned document contains modified and incorrect information. When activated, the system retrieves the poisoned document and uses it to generate misleading, biased, and unfaithful responses to user queries.

Poison Attacks and Adversarial Prompts Against an Informed University Virtual Assistant

Evaluated models: Barkplug V.2

SQL Injection Jailbreak

Published Nov 3, 2024Analyzed Dec 28, 2024Source: arXiv

A novel SQL Injection Jailbreak (SIJ) vulnerability allows attackers to bypass safety mechanisms in Large Language Models (LLMs) by manipulating the structure of input prompts. The attack leverages the model's processing of system prompts, user prefixes, user prompts, and assistant prefixes to effectively "comment out" the expected response prefix and inject harmful instructions, causing the LLM to generate unsafe content. This vulnerability exploits the external properties of the LLM…

SQL Injection Jailbreak: a structural disaster of large language models

Evaluated models: DeepSeek LLM 7B Chat, Llama 2 7B Chat, Llama 3.1 8B Instruct+2 more

DeepSeek LLM 7B Chat
Llama 2 7B Chat
Llama 3.1 8B Instruct
Mistral 7B Instruct v0.2
Vicuna 7B v1.5

Nonlinear Prompt Jailbreak Features

Published Nov 2, 2024Analyzed Dec 28, 2024Source: arXiv

Large language models (LLMs) are vulnerable to jailbreak attacks exploiting nonlinear features within prompt encodings. These features, not detectable by linear methods, allow adversaries to reliably elicit harmful outputs despite safety training. Different attack methods leverage distinct nonlinear features, limiting the transferability of detection and mitigation techniques.

What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks

Evaluated models: Gemma 7B IT, Llama 3 8B Instruct

Composable String Jailbreaks

Published Nov 1, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to jailbreaking attacks using sequences of invertible string transformations (string compositions). Attackers can combine multiple transformations (e.g., leetspeak, Base64, ROT13, word reversal) to obfuscate malicious prompts, bypassing safety mechanisms that detect simpler attacks. Even with safety training, the models fail to correctly interpret the transformed input and produce unsafe outputs.

Plentiful Jailbreaks with String Compositions

Evaluated models: Claude 3 Haiku, Claude 3 Opus, Claude 3.5 Sonnet+2 more

Claude 3 Haiku
Claude 3 Opus
Claude 3.5 Sonnet
GPT-4o
GPT-4o Mini

Emoji Judge Bypass

Published Nov 1, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) used as safety judges are vulnerable to an "Emoji Attack," a prompt injection technique that leverages token segmentation bias. Inserting emojis within tokens alters sub-token embeddings, misleading the judge LLM into classifying harmful content as safe. The attack's effectiveness is amplified by strategically placing emojis to maximize the embedding discrepancy between sub-tokens and the original token.

Emoji Attack: A Method for Misleading Judge LLMs in Safety Risk Detection

Evaluated models: GPT-3.5 Turbo, GPT-4, Llama Guard+3 more

GPT-3.5 Turbo
GPT-4
Llama Guard
Llama Guard 2
ShieldLM
WildGuard

Ensemble Black-box Jailbreak

Published Oct 31, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to transferable ensemble black-box jailbreak attacks. The vulnerability allows an attacker to bypass safety mechanisms and elicit undesired or harmful responses from the LLM by using an ensemble of LLM-as-attacker methods that optimize malicious prompts, adaptively adjusting resources based on prompt difficulty, and strategically modifying prompt semantics to evade detection.

Transferable Ensemble Black-box Jailbreak Attacks on Large Language Models

Evaluated models: Deepseek-v2.5, Gemma 2B IT, Gemma 2 9B IT+5 more

Deepseek-v2.5
Gemma 2B IT
Gemma 2 9B IT
GLM 4 Plus
Glm-4-flash
GPT-4
Llama 3 8B Instruct
Qwen-max-latest

Benign Mirroring Jailbreak

Published Oct 28, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to stealthy jailbreak attacks leveraging benign data mirroring. Attackers train a local "mirror model" on benign data obtained from the target LLM. This mirror model, mimicking the target's behavior, is then used to generate adversarial prompts, which are subsequently deployed against the target LLM, bypassing content moderation systems due to the lack of overtly malicious content in the initial data gathering phase.

Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring

Evaluated models: GPT-3.5 Turbo, GPT-4o Mini, Llama 2 Chat+1 more

GPT-3.5 Turbo
GPT-4o Mini
Llama 2 Chat
Llama 3 8B Instruct

SMILES-Prompting LLM Jailbreak

Published Oct 21, 2024Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) used in chemical synthesis applications are vulnerable to a novel attack vector, dubbed "SMILES-prompting," which leverages the Simplified Molecular-Input Line-Entry System (SMILES) notation to bypass safety mechanisms and elicit instructions for synthesizing hazardous substances. The attack exploits the LLM's inability to effectively filter or interpret SMILES strings representing dangerous chemicals, leading to the disclosure of synthesis procedures.

SMILES-Prompting: A Novel Approach to LLM Jailbreak Attacks in Chemical Synthesis

Evaluated models: GPT-4o, Llama 3 70B Instruct

Related research categories