Attack Surface

Prompt layer Research

Security research involving prompt handling and processing

610 matching entries

Prompt layer research results

610 entries

Personalized Encryption Jailbreak

Published Feb 26, 2024Analyzed Dec 29, 2024Source: arXiv

A vulnerability exists in several Large Language Models (LLMs) allowing attackers to bypass safety and ethical protocols through a novel code injection technique using personalized encryption and decryption functions. The attack leverages the LLMs' code execution capabilities to process encrypted malicious instructions, circumventing the intent security recognition mechanism.

Codechameleon: Personalized encryption framework for jailbreaking large language models

Evaluated models: GPT-3.5 Turbo, GPT-4-1106, Llama 2 13B Chat+4 more

GPT-3.5 Turbo
GPT-4-1106
Llama 2 13B Chat
Llama 2 70B Chat
Llama 2 7B Chat
Vicuna 13B
Vicuna 7B

Rainbow Teaming LLM Jailbreak

Published Feb 26, 2024Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to adversarial prompts generated by the Rainbow Teaming technique. Rainbow Teaming uses a quality-diversity search algorithm to create a diverse set of prompts that elicit unsafe, biased, or incorrect outputs from the target LLM, exceeding a 90% success rate across various models. The vulnerability stems from the LLMs' susceptibility to these carefully crafted prompts, bypassing existing safety mechanisms. These prompts are highly transferable across…

Rainbow teaming: Open-ended generation of diverse adversarial prompts

Evaluated models: Codellama 34B Instruct, CodeLlama 7B Instruct, GPT-4+6 more

Codellama 34B Instruct
CodeLlama 7B Instruct
GPT-4
Llama 2 13B Chat
Llama 2 70B Chat
Llama 2 7B Chat
Llama 3-instruct 8B
Mistral 7B
Vicuna 7B v1.5

Prompt Decomposition Jailbreak

Published Feb 25, 2024Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to DrAttack, a jailbreaking technique that decomposes malicious prompts into semantically neutral sub-prompts. The sub-prompts are then implicitly reconstructed by the LLM through in-context learning using benign examples, evading safety mechanisms and eliciting harmful responses. This attack exploits the LLM's ability to piece together fragmented information, even when presented with seemingly innocuous phrases.

Drattack: Prompt decomposition and reconstruction makes powerful llm jailbreakers

Evaluated models: Claude 1, Claude 2, Gemini Pro+6 more

Claude 1
Claude 2
Gemini Pro
GPT-3.5 Turbo
GPT-4
Llama 2 13B
Llama 2 7B
Vicuna 13B
Vicuna 7B

Cognitive Consistency Jailbreak

Published Feb 24, 2024Analyzed Dec 29, 2024Source: arXiv

A vulnerability in several large language models (LLMs) allows attackers to bypass safety restrictions ("jailbreaking") by employing a Foot-in-the-Door (FITD) technique. This involves progressively escalating prompts, starting with innocuous requests and gradually leading to the elicitation of harmful or restricted information. The LLM's tendency towards cognitive consistency makes it more likely to respond to subsequent, increasingly sensitive prompts after initially agreeing to less harmful…

Foot In The Door: Understanding Large Language Model Jailbreaking via Cognitive Psychology

Evaluated models: Chatglm-2 (chatglm2-6B), Chatglm-3 (chatglm3-6B), Claude 2.1+5 more

Chatglm-2 (chatglm2-6B)
Chatglm-3 (chatglm3-6B)
Claude 2.1
Claude Instant 1.2
Gemini (gemini-pro)
GPT-3.5 (GPT-3.5-turbo-1106)
GPT-4 (GPT-4-1106-preview)
Llama-2 (llama2-7B-chat)

Universal Guardrail Bypass

Published Feb 24, 2024Analyzed Dec 29, 2024Source: arXiv

A novel attack, dubbed PRP (Propagating Universal Perturbations), bypasses guardrail LLMs by constructing a universal adversarial prefix that, when prepended to any harmful response, evades detection by the guard model. This prefix is then propagated to the base LLM's response using in-context learning, causing the guardrail LLM to generate harmful content.

Prp: Propagating universal perturbations to attack large language model guard-rails

Evaluated models: Gemini Pro, GPT 3.5-turbo-0125, Guanaco 13B+5 more

Gemini Pro
GPT 3.5-turbo-0125
Guanaco 13B
Llama 2 70B Chat
Mistral 7B Instruct
Vicuna-33B-v1.3
Wizard-lm-falcon-7B-uncensored
Wizardlm7B-uncensored

Semantic Mirror Jailbreak

Published Feb 21, 2024Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a novel semantic mirror jailbreak attack. This attack leverages a genetic algorithm to generate jailbreak prompts that are semantically similar to benign prompts, evading defenses based on semantic similarity metrics. The attack achieves this by optimizing for both semantic similarity to the original question and the ability to elicit harmful responses.

Semantic Mirror Jailbreak: Genetic Algorithm Based Jailbreak Prompts Against Open-source LLMs

Evaluated models: Guanaco 7B, Llama 2 7B Chat, Vicuna 7B

ASCII Art Jailbreak

Published Feb 19, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) exhibit vulnerability to a novel jailbreak attack, "ArtPrompt," which leverages the models' poor ability to recognize ASCII art representations of words. By replacing sensitive words in a prompt with their ASCII art equivalents, the attacker bypasses safety filters designed to prevent the generation of harmful content.

Artprompt: Ascii art-based jailbreak attacks against aligned llms

Evaluated models: GPT-3.5 Turbo, GPT-4

Complex Cipher Jailbreak

Published Feb 16, 2024Analyzed Mar 4, 2025Source: arXiv

Large Language Models (LLMs) with advanced reasoning capabilities are vulnerable to jailbreaking attacks using novel, complex, and layered custom encryption schemes. LLMs' ability to decipher these ciphers, exceeding the capabilities of less sophisticated models, enables attackers to bypass existing safety mechanisms by encoding malicious prompts.

When" Competency" in Reasoning Opens the Door to Vulnerability: Jailbreaking LLMs via Novel Complex Ciphers

Evaluated models: Gemini 1.5 Flash, GPT-4o, Llama 3.1 70B Instruct+1 more

Gemini 1.5 Flash
GPT-4o
Llama 3.1 70B Instruct
Llama 3.1 8B Instruct

Implicit Clue Jailbreak

Published Feb 14, 2024Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to an indirect jailbreak attack, termed "Puzzler," which leverages implicit clues instead of explicit malicious intent in prompts. By providing associated behaviors or hints related to a malicious query, Puzzler elicits malicious responses from the LLM, bypassing its safety mechanisms. The attack works by first obtaining "defensive measures" from the LLM against a target malicious action, then querying for the corresponding "offensive measures" that…

Play guessing game with llm: Indirect jailbreak attack with implicit clues

Evaluated models: Gemini Pro, GPT-3.5 Turbo, GPT-4+3 more

Gemini Pro
GPT-3.5 Turbo
GPT-4
GPT-4 Turbo
Llama 13B
Llama 7B

Multi-Turn Contextual Jailbreak

Published Feb 14, 2024Analyzed Dec 29, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a multi-round "Contextual Interaction Attack" where a series of benign preliminary questions, crafted to be semantically aligned with a malicious target query, are used to manipulate the LLM's context vector. The autoregressive nature of LLMs causes them to incorporate previous conversation rounds into their generation process, allowing the attacker to prime the model into providing harmful information in response to the final, seemingly benign…

Leveraging the Context through Multi-Round Interactions for Jailbreaking Attacks

Evaluated models: Claude 2, GPT-3.5 Turbo, GPT-4+3 more

Claude 2
GPT-3.5 Turbo
GPT-4
Llama 2 7B
Mixtral 8x7B
Vicuna 7B

Related research categories