Attack Surface

Prompt layer Research

Security research involving prompt handling and processing

610 matching entries

Prompt layer research results

610 entries

Custom GPT Prompt Injection

Published Nov 20, 2023Analyzed Dec 28, 2024Source: arXiv

A prompt injection vulnerability in OpenAI's custom GPT models allows attackers to extract the system prompt and potentially leak user-uploaded files. Attackers craft malicious prompts that manipulate the LLM into revealing sensitive information, even when defensive prompts are in place. The vulnerability is exacerbated when the model includes a code interpreter.

Assessing prompt injection risks in 200+ custom gpts

Evaluated models: Not reported

LLM Cipher Jailbreak

Published Aug 12, 2023Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) such as GPT-4, while employing safety alignment techniques, exhibit vulnerability to "CipherChat" attacks. CipherChat leverages cipher prompts (e.g., ASCII, Unicode, Caesar cipher, Morse code) combined with system role descriptions and few-shot enciphered demonstrations to bypass safety mechanisms trained on natural language. This allows an attacker to elicit unsafe responses from the LLM, effectively evading safety filters. The vulnerability is amplified by the…

Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher

Evaluated models: Claude 2, Falcon-chat-180B, GPT-3.5+5 more

Claude 2
Falcon-chat-180B
GPT-3.5
GPT-3.5 Turbo
GPT-4
Llama2-chat-13B
Llama-2-chat-70B
Llama2-chat-7B

Color-Aware Watermark Bypass

Published Mar 19, 2024Analyzed Dec 28, 2024Source: arXiv

A color-aware attack, Self Color Testing-based Substitution (SCTS), bypasses watermarking mechanisms in LLMs designed to identify AI-generated text. SCTS exploits the LLM's compliance with instructions to infer the "color" (green/red token classification) of tokens, allowing for targeted substitution of watermarked tokens with non-watermarked tokens, thus evading watermark detection. The attack is particularly effective against watermarks that utilize logit perturbation to bias token selection.

Bypassing LLM Watermarks with Color-Aware Substitutions

Evaluated models: Not reported

Adversarial Code Generation

Published Dec 7, 2023Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) used for code generation are vulnerable to adversarial natural language instructions that preserve semantic meaning but induce the generation of functionally correct code containing specific vulnerabilities. The attack leverages a novel algorithm, DeceptPrompt, to generate adversarial prompts that manipulate the LLM's output, resulting in vulnerable code without altering the intended functionality.

Deceptprompt: Exploiting llm-driven code generation via adversarial natural language instructions

Evaluated models: Code Llama 7B, StarChat 15B, WizardCoder 15B+1 more

Code Llama 7B
StarChat 15B
WizardCoder 15B
WizardCoder 3B

Hidden Prompt Injection Attacks

Published Oct 16, 2023Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to Compositional Instruction Attacks (CIA), where malicious prompts are embedded within seemingly harmless instructions. This allows attackers to bypass safety mechanisms and elicit harmful responses from the model, even if the individual components of the prompt would be flagged as safe. The attack exploits the model's inability to correctly identify underlying malicious intent within composite instructions.

Prompt packer: Deceiving llms through compositional instruction with hidden attacks

Evaluated models: ChatGLM2 6B, GPT-3.5 Turbo, GPT-4

Rainbow Teaming LLM Jailbreak

Published Feb 26, 2024Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to adversarial prompts generated by the Rainbow Teaming technique. Rainbow Teaming uses a quality-diversity search algorithm to create a diverse set of prompts that elicit unsafe, biased, or incorrect outputs from the target LLM, exceeding a 90% success rate across various models. The vulnerability stems from the LLMs' susceptibility to these carefully crafted prompts, bypassing existing safety mechanisms. These prompts are highly transferable across…

Rainbow teaming: Open-ended generation of diverse adversarial prompts

Evaluated models: Codellama 34B Instruct, CodeLlama 7B Instruct, GPT-4+6 more

Codellama 34B Instruct
CodeLlama 7B Instruct
GPT-4
Llama 2 13B Chat
Llama 2 70B Chat
Llama 2 7B Chat
Llama 3-instruct 8B
Mistral 7B
Vicuna 7B v1.5

Chain-of-Utterance Jailbreak

Published Aug 18, 2023Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a "Chain of Utterances" (CoU) based prompt injection attack. This attack exploits the LLM's ability to engage in multi-turn conversations and role-playing, tricking it into providing harmful or unsafe responses even when presented with safety guidelines. The attack leverages a crafted conversation between two agents ("Red-LM," a malicious agent, and "Base-LM," a seemingly helpful agent) to elicit unethical responses from the Base-LM by subtly…

Red-teaming large language models using chain of utterances for safety-alignment

Evaluated models: Not reported

Code-Switching LLM Jailbreak

Published Jun 17, 2024Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) exhibit increased vulnerability to adversarial prompts employing code-switching techniques, where multiple languages are interwoven within a single query. This vulnerability stems from an unintended correlation between the resource availability of the languages used in the prompt and the LLM's safety alignment. LLMs trained on imbalanced multilingual data are more susceptible to attacks leveraging low-resource languages, resulting in a higher rate of unsafe or…

Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understanding

Evaluated models: Not reported

LLM Red Teaming Framework

Published Oct 19, 2023Analyzed Dec 28, 2024Source: arXiv

A vulnerability in large language models (LLMs) allows attackers to craft malicious prompts that induce the LLM to generate harmful content, such as fraudulent material, racist remarks, or instructions for illegal activities. The vulnerability arises from the LLM's inability to reliably distinguish between benign and malicious instructions disguised within seemingly innocuous prompts. Attackers can exploit this by leveraging techniques like obfuscation, code injection/payload splitting, and…

Attack prompt generation for red teaming and defending large language models

Evaluated models: Not reported

RL-Based LLM Privacy Leak

Published Dec 7, 2024Analyzed Dec 28, 2024Source: arXiv

Large Language Models (LLMs) are vulnerable to a novel agentic-based red-teaming attack, PrivAgent, which uses reinforcement learning to generate adversarial prompts. These prompts can extract sensitive information, including system prompts and portions of training data, from target LLMs even with existing guardrail defenses. The attack leverages a custom reward function based on a normalized sliding-window word edit similarity metric to guide the learning process, enabling it to overcome the…

PrivAgent: Agentic-based Red-teaming for LLM Privacy Leakage

Evaluated models: Not reported

Related research categories