Attack Surface

Prompt layer Research

Security research involving prompt handling and processing

610 matching entries

Prompt layer research results

610 entries

Intent Flattening Jailbreak

Published Feb 5, 2025Analyzed Dec 9, 2025Source: arXiv

A vulnerability exists in the safety alignment mechanisms of Large Language Models (LLMs) related to the model's intent perception capabilities. The specific attack vector, termed "Perceived-importance Flatten" (PiF), circumvents safety guardrails by modifying neutral-intent tokens within a malicious prompt using synonym replacement. Unlike traditional jailbreak attacks that rely on appending lengthy, high-perplexity adversarial suffixes (which suffer from distributional dependency and often…

Understanding and Enhancing the Transferability of Jailbreaking Attacks

Evaluated models: Llama 2 13B Chat, Llama 3.1 8B Instruct, Mistral 7B Instruct+5 more

Llama 2 13B Chat
Llama 3.1 8B Instruct
Mistral 7B Instruct
Vicuna 13B v1.5
GPT-4 0613
o1-preview
Claude 3.5 Sonnet
Gemini 1.5 Flash

LLM RAG Decoy Overthink

Published Feb 4, 2025Analyzed Dec 9, 2025Source: arXiv

A resource exhaustion and algorithmic complexity vulnerability exists in applications utilizing Reasoning Large Language Models (e.g., OpenAI o1, DeepSeek R1) that process untrusted external context (such as Retrieval-Augmented Generation systems). The vulnerability, dubbed "OverThink," allows an attacker to perform an indirect prompt injection by embedding "decoy" reasoning problems—specifically computation-intensive tasks like Sudoku puzzles or Markov Decision Processes (MDPs)—into the…

Overthink: Slowdown attacks on reasoning llms

Evaluated models: o1, o3, DeepSeek R1

AP-Test Guardrail Identification

Published Feb 3, 2025Analyzed Mar 4, 2025Source: arXiv

This vulnerability allows attackers to identify the presence and location (input or output stage) of specific guardrails implemented in Large Language Models (LLMs) by using carefully crafted adversarial prompts. The attack, termed AP-Test, leverages a tailored loss function to optimize these prompts, maximizing the likelihood of triggering a specific guardrail while minimizing triggering others. Successful identification provides attackers with valuable information to design more effective…

Peering Behind the Shield: Guardrail Identification in Large Language Models

Evaluated models: Aegis Defensive, Aegis Permissive, GPT-4o+8 more

Aegis Defensive
Aegis Permissive
GPT-4o
Llama Guard
Llama Guard 2
Llama Guard 3
Perspective
ShieldGemma 2B
ShieldGemma 9B
ShieldGemma 27B
WildGuard

Adversarial LLM Jailbreak

Published Feb 3, 2025Analyzed Feb 16, 2025Source: arXiv

A vulnerability in Large Language Models (LLMs) allows adversarial reasoning attacks to bypass safety mechanisms and elicit harmful responses. The vulnerability stems from the insufficient robustness of existing LLM safety measures against iterative prompt refinement guided by a loss function that measures the LLM's proximity to generating a target harmful response. This allows an attacker to effectively navigate the prompt space, even against adversarially trained models, resulting in…

Adversarial Reasoning at Jailbreaking Time

Evaluated models: Claude 3.5 Sonnet, Cygnet, Gemini 1.5 Pro+8 more

Claude 3.5 Sonnet
Cygnet
Gemini 1.5 Pro
GPT-4
Llama 2 7B
Llama 3 8B
Llama 3.1 405B
Mixtral 8x7B
o1-preview
R2D2
Vicuna 13B v1.5

Inherited GPT Policy Violations

Published Feb 3, 2025Analyzed Dec 9, 2025Source: arXiv

A policy compliance vulnerability exists in the OpenAI GPT Store ecosystem affecting Custom GPTs. The vulnerability stems from the inheritance of safety alignment weaknesses from foundational models (GPT-4 and GPT-4o) and the insufficient enforcement of usage policies during the customization and review process. Custom GPTs can be trivially manipulated to violate safety guidelines—specifically regarding Cybersecurity (malware generation), Academic Integrity (ghostwriting), and Romantic…

Towards Safer Chatbots: A Framework for Policy Compliance Evaluation of Custom GPTs

Evaluated models: GPT-4, GPT-4o

Weight Tampering Unlearning Bypass

Published Feb 3, 2025Analyzed Dec 9, 2025Source: arXiv

State-of-the-art machine unlearning and safety fine-tuning methods for Large Language Models (LLMs) fail to robustly remove hazardous capabilities or refusal mechanisms from model weights. While these methods suppress model outputs during standard input-output interactions, the underlying capabilities remain latent in the parameter space. An attacker with access to model weights (e.g., via open releases or leaked weights) can restore "unlearned" knowledge (such as dual-use biology hazards) or…

Model tampering attacks enable more rigorous evaluations of llm capabilities

Evaluated models: Llama 3 8B

Multimodal Flanking Jailbreak

Published Feb 2, 2025Analyzed Mar 4, 2025Source: arXiv

A novel "Flanking Attack" exploits the vulnerability of multimodal LLMs (e.g., Google Gemini) to bypass content moderation filters by embedding adversarial prompts within a sequence of benign prompts. The attack leverages the LLM's processing of both audio and text, obfuscating harmful requests through contextualization and layering, thereby yielding policy-violating responses.

From Compliance to Exploitation: Jailbreak Prompt Attacks on Multimodal LLMs

Evaluated models: Not reported

Unlearning Robustness Gap

Published Feb 1, 2025Analyzed Jan 14, 2026Source: arXiv

Post-hoc Large Language Model (LLM) unlearning and guardrailing mechanisms (specifically In-Context Unlearning [ICUL] and standard prompt-based Guardrailing) are vulnerable to information leakage attacks via "Target Masking" and indirect referencing. These systems rely on superficial semantic matching to suppress "forget sets" (specific entities or concepts). Attackers can bypass these restrictions by querying associated properties, relationships, or pseudonyms rather than the explicit target…

Alu: Agentic llm unlearning

Evaluated models: GPT-4o, Llama 2 7B, Llama 3.2 3B+2 more

GPT-4o
Llama 2 7B
Llama 3.2 3B
Qwen 2.5 14B
Phi-3

Sophisticated Reasoning Bypass

Published Jan 31, 2025Analyzed Dec 30, 2025Source: arXiv

Large Language Models (LLMs), specifically instruction-following models using standard refusal training and adversarial training (such as Llama-3.1-8B-Instruct and Mistral-7B-V0.2), contain a vulnerability related to safety alignment bypass. The vulnerability arises from the models' inability to generalize safety reasoning to Out-Of-Distribution (OOD) inputs and scenarios involving competing objectives. Attackers can exploit this by employing linguistic manipulation (slang, uncommon dialects…

Enhancing Model Defense Against Jailbreaks with Proactive Safety Reasoning

Evaluated models: Llama 3.1 8B Instruct, Mistral 7B Instruct v0.2

Guardrail Bypass Harmful Fine-tuning

Published Jan 29, 2025Analyzed Mar 19, 2025Source: arXiv

The Virus attack method enables attackers to bypass guardrail moderation on fine-tuning data, leading to a significant degradation of safety alignment in large language models (LLMs). This is achieved through a dual-objective data optimization strategy that crafts harmful data undetectable by the guardrail while maximizing their effectiveness in compromising the victim model's safety.

Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation

Evaluated models: Llama 3 8B, Llama Guard 2

Related research categories