Attack Surface

Prompt layer Research

Security research involving prompt handling and processing

610 matching entries

Prompt layer research results

610 entries

Self-Instruct LLM Jailbreak

Published 1/1/2025

Analyzed 1/26/2025

Large Language Models (LLMs) are vulnerable to a self-instruct few-shot jailbreaking attack that leverages pattern and behavior learning to bypass safety mechanisms. The attack efficiently induces harmful outputs by injecting a strategically chosen response prefix into the model's prompt and exploiting the model's tendency to mimic co-occurrence patterns of special tokens preceding the prefix. This allows the attacker to elicit unsafe responses with a small number of carefully crafted…

Self-Instruct Few-Shot Jailbreaking: Decompose the Attack into Pattern and Behavior Learning

Evaluated models: GPT-2, Llama 2 7B Chat, Llama 3 8B Instruct +7 more

Source: arXiv

Sophisticated Reasoning Bypass

Published 1/1/2025

Analyzed 12/30/2025

Large Language Models (LLMs), specifically instruction-following models using standard refusal training and adversarial training (such as Llama-3.1-8B-Instruct and Mistral-7B-V0.2), contain a vulnerability related to safety alignment bypass. The vulnerability arises from the models' inability to generalize safety reasoning to Out-Of-Distribution (OOD) inputs and scenarios involving competing objectives. Attackers can exploit this by employing linguistic manipulation (slang, uncommon dialects…

Enhancing Model Defense Against Jailbreaks with Proactive Safety Reasoning

Evaluated models: Llama 3.1 8B Instruct, Mistral 7B Instruct v0.2

Source: arXiv

Targeted Text-Diffusion Jailbreak

Published 1/1/2025

Analyzed 3/19/2025

Large language models (LLMs) are vulnerable to adversarial prompt engineering attacks that leverage proximity constraints to elicit harmful behaviors. By subtly modifying benign prompts within a semantically close embedding space, attackers can bypass existing safety mechanisms and induce undesired outputs, even when the original prompts would not trigger such a response. This vulnerability exploits the model's sensitivity to small perturbations in the input embedding, resulting in the…

Text-Diffusion Red-Teaming of Large Language Models: Unveiling Harmful Behaviors with Proximity Constraints

Evaluated models: GPT-2, Llama 2 7B Chat, Vicuna 7B

Source: arXiv

Task-in-Prompt Jailbreak

Published 1/1/2025

Analyzed 12/9/2025

Large Language Models (LLMs) including GPT-4o, LLaMA 3.2, and others exhibit a vulnerability to "Task-in-Prompt" (TIP) adversarial attacks. This vulnerability allows attackers to bypass safety alignment and content filtering mechanisms by embedding prohibited instructions within benign sequence-to-sequence tasks (such as ciphers, riddles, code execution, or text transformation). The model implicitly decodes the obfuscated content via self-attention mechanisms during token generation…

The TIP of the Iceberg: Revealing a Hidden Class of Task-in-Prompt Adversarial Attacks on LLMs

Evaluated models: Not reported

Source: arXiv

Alignment-Based LLM Jailbreak

Published 12/1/2024

Analyzed 1/26/2025

Large Language Models (LLMs) employing reinforcement learning from human feedback (RLHF) for safety alignment are vulnerable to a novel "alignment-based" jailbreak attack. This attack leverages a best-of-N sampling approach with an adversarial LLM to efficiently generate prompts that bypass safety mechanisms and elicit unsafe responses from the target LLM, without requiring additional training or access to the target LLM's internal parameters. The attack exploits the inherent tension between…

LIAR: Leveraging Alignment (Best-of-N) to Jailbreak LLMs in Seconds

Evaluated models: Falcon 7B, GPT-2, Llama 3.1 8B +6 more

Source: arXiv

BarkPlug Data Poisoning Attack

Published 12/1/2024

Analyzed 3/19/2025

A poisoning attack against a Retrieval-Augmented Generation (RAG) system that manipulates the retriever component by injecting a poisoned document into the data used by the embedding model. This poisoned document contains modified and incorrect information. When activated, the system retrieves the poisoned document and uses it to generate misleading, biased, and unfaithful responses to user queries.

Poison Attacks and Adversarial Prompts Against an Informed University Virtual Assistant

Evaluated models: Barkplug V.2

Source: arXiv

Best-of-N Prompt Augmentation

Published 12/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) across multiple modalities (text, vision, audio) are vulnerable to a "Best-of-N" (BoN) jailbreaking attack. This attack repeatedly submits slightly modified versions of a harmful prompt (e.g., text with altered capitalization, images with modified text style, audio with altered pitch or speed) until a safety mechanism is bypassed and a harmful response is elicited. The effectiveness of the attack scales with the number of attempts (N). While individual…

Best-of-N Jailbreaking

Evaluated models: Claude 3 Opus, Claude 3.5 Sonnet, Cygnet +10 more

Source: arXiv

Contextual Adversarial Prompts

Published 12/1/2024

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to human-readable adversarial prompts crafted using situational context derived from movie scripts. These prompts, which combine a malicious prompt, a seemingly innocuous adversarial insertion, and relevant contextual information, can bypass LLMs' safety mechanisms and elicit harmful responses. The technique leverages the LLM's ability to understand context and generate responses consistent with that context to mask the malicious intent. The…

Human-Readable Adversarial Prompts: An Investigation into LLM Vulnerabilities Using Situational Context

Evaluated models: Flan-t5 Large, Gemini 1.5 Pro, Gemma 2B IT +16 more

Source: arXiv

Diffusion-Driven LLM Jailbreak

Published 12/1/2024

Analyzed 12/28/2024

DiffusionAttacker exploits a vulnerability in Large Language Models (LLMs) allowing manipulation of prompts to elicit harmful responses, even when the model incorporates safety mechanisms. The attack leverages a sequence-to-sequence diffusion model to rewrite harmful prompts, making them appear harmless to the LLM's internal representation while preserving their original semantic meaning. This bypasses safety filters and elicits undesired outputs.

DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak

Evaluated models: Alpaca 7B, Claude 3.5 Sonnet, GPT-3.5 Turbo +4 more

Source: arXiv

Linked-Task LLM Jailbreak

Published 12/1/2024

Analyzed 12/28/2024

A novel jailbreak paradigm, Simple Assistive Task Linkage (SATA), circumvents LLM safeguards by masking harmful keywords in a malicious query and using a secondary, simple assistive task (e.g., masked language modeling or element lookup by position) to convey the masked keywords' semantics to the LLM. This distracts the LLM and allows it to bypass safety checks, leading to the generation of harmful responses.

SATA: A Paradigm for LLM Jailbreak via Simple Assistive Task Linkage

Evaluated models: Claude-v2, GPT-3.5 Turbo, GPT-4o +3 more

Source: arXiv

Related research categories