Attack Surface

Prompt layer Research

Security research involving prompt handling and processing

610 matching entries

Prompt layer research results

610 entries

Arabizi LLM Jailbreak

Published Jun 26, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) exhibit vulnerability to jailbreak attacks when prompted using Arabic transliteration and Arabizi (Arabic chatspeak). While LLMs demonstrate robustness to standard Arabic prompts, even with prefix injection, the use of transliterated or Arabizi prompts bypasses safety mechanisms, leading to the generation of unsafe content. This is due to the model's learned associations with specific words in these non-standard forms, which differ from its understanding of the…

Jailbreaking llms with arabic transliteration and arabizi

Evaluated models: Anthropic Claude-3-sonnet20240229, GPT-4o, Llama2-7-billion+2 more

Anthropic Claude-3-sonnet20240229
GPT-4o
Llama2-7-billion
Openai GPT-3.5-turbo-0125
Openai GPT-4-0613

Prompt layer

Jailbreak

Black-box

Obscured Prompt Jailbreak

Published Jun 19, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to jailbreaking attacks using "obscure" input prompts. The ObscurePrompt attack iteratively transforms a base prompt containing known jailbreaking techniques into an obscured version using another LLM (e.g., GPT-4). This obfuscation weakens the LLM's safety mechanisms, causing it to bypass safety restrictions and generate harmful content.

Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings

Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4o Mini+5 more

GPT-3.5 Turbo
GPT-4
GPT-4o Mini
Llama 2 7B
Llama 2 70B
Llama 3 8B
Llama 3 70B
Vicuna 7B

LLM Copyright Jailbreak

Published Jun 18, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to prompt injection attacks that can bypass their internal copyright compliance mechanisms, causing them to generate verbatim copyrighted text. The vulnerability stems from insufficient robustness against prompt engineering techniques that manipulate the model into ignoring or circumventing its safety filters designed for copyright protection.

SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generation

Evaluated models: Claude 3 Haiku, Gemini 1.5 Pro, Gemini Pro+5 more

Claude 3 Haiku
Gemini 1.5 Pro
Gemini Pro
GPT-3.5 Turbo
GPT-4o
Llama 2 7B Chat
Llama 3 8B Instruct
Mistral 7B Instruct

Chat Template Jailbreak

Published Jun 17, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) fine-tuned using chat templates are vulnerable to ChatBug, allowing malicious actors to bypass safety mechanisms by crafting prompts that intentionally deviate from the expected template format or overflow message fields. This exploits the LLM’s reliance on the template structure without enforcing similar constraints on user input.

ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates

Evaluated models: Claude 2.1, GPT-3.5 Turbo

Code-Switching LLM Jailbreak

Published Jun 17, 2024Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) exhibit increased vulnerability to adversarial prompts employing code-switching techniques, where multiple languages are interwoven within a single query. This vulnerability stems from an unintended correlation between the resource availability of the languages used in the prompt and the LLM's safety alignment. LLMs trained on imbalanced multilingual data are more susceptible to attacks leveraging low-resource languages, resulting in a higher rate of unsafe or…

Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understanding

Evaluated models: Not reported

Knowledge-Based LLM Jailbreak

Published Jun 17, 2024Analyzed Mar 4, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to knowledge-based jailbreaks, where an attacker provides domain-specific knowledge to elicit harmful or unintended outputs. The vulnerability stems from the LLM's ability to process and respond to knowledge inputs in a way that circumvents safety mechanisms, even if the input knowledge itself isn't inherently malicious. Attackers leverage this by constructing prompts that combine seemingly innocuous knowledge with subtly manipulative phrasing to…

Knowledge-to-jailbreak: One knowledge point worth one attack

Evaluated models: FinanceChat 7B, GPT-3.5 Turbo, GPT-4 Turbo+6 more

FinanceChat 7B
GPT-3.5 Turbo
GPT-4 Turbo
LawChat 7B
Llama 2 13B Chat
Llama 2 7B
Llama 2 7B Chat
Mistral 7B Instruct
Vicuna 7B v1.5

DRL-Guided LLM Jailbreak

Published Jun 13, 2024Analyzed Dec 29, 2024Source: arXiv

A deep reinforcement learning (DRL) based attack, termed RLbreaker, demonstrates the ability to more efficiently generate jailbreaking prompts for large language models (LLMs) than existing methods. The attack leverages a DRL agent to guide the search for effective prompt structures, bypassing safety mechanisms and eliciting undesirable responses to harmful questions. The effectiveness stems from the DRL agent's ability to strategically select prompt mutators, rather than relying on random…

When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided Search

Evaluated models: GPT-3.5 Turbo, Llama 2 70B Chat, Llama 2 7B Chat+3 more

GPT-3.5 Turbo
Llama 2 70B Chat
Llama 2 7B Chat
Mixtral 8x7B Instruct
Vicuna 13B
Vicuna 7B

Hidden Structure Jailbreak

Published Jun 13, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to jailbreak attacks exploiting uncommon text-encoded structures (UTES) rarely encountered during training. These UTES, such as JSON, tree representations, or LaTeX code, embedded within prompts, cause LLMs to bypass safety mechanisms and generate harmful content. The attack's success stems from the LLM's difficulty in processing and interpreting these unusual structures, coupled with the obfuscation of malicious instructions within the structured…

StructuralSleight: Automated Jailbreak Attacks on Large Language Models Utilizing Uncommon Text-Encoded Structure

Evaluated models: Claude 2, Claude 3 Opus, GPT-3.5 Turbo+3 more

Claude 2
Claude 3 Opus
GPT-3.5 Turbo
GPT-4
GPT-4o
Llama 3 70B

RL-Powered LLM Jailbreak

Published Jun 13, 2024Analyzed Dec 29, 2024Source: arXiv

RL-JACK is a reinforcement learning-based black-box attack that generates jailbreaking prompts to bypass safety mechanisms in LLMs. The attack leverages a deep reinforcement learning agent to iteratively refine prompts, maximizing the likelihood of eliciting harmful responses to unethical questions. The effectiveness stems from a novel reward function that provides continuous feedback based on cosine similarity to a reference answer from an unaligned LLM, and an action space that strategically…

RL-JACK: Reinforcement Learning-powered Black-box Jailbreaking Attack against LLMs

Evaluated models: Falcon-40B-instruct, GPT-3.5 Turbo, Llama 2 70B Chat+3 more

Falcon-40B-instruct
GPT-3.5 Turbo
Llama 2 70B Chat
Llama 2 7B Chat
Vicuna 13B
Vicuna 7B

Bi-Modal Adversarial Jailbreak

Published Jun 6, 2024Analyzed Dec 28, 2024Source: arXiv

Large Vision Language Models (LVLMs) are vulnerable to a bi-modal adversarial prompt attack (BAP). BAP leverages a combined textual and visual prompt to bypass safety mechanisms and elicit harmful responses, even in models designed to resist single-modality attacks. The attack first introduces a query-agnostic adversarial perturbation to the visual prompt, making the model more likely to respond positively regardless of the text. Then, an LLM refines the textual prompt iteratively to achieve…

Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt

Evaluated models: Not reported

Related research categories