Attack Type

Injection Research

Attacks that inject malicious content into model inputs

337 matching entries

Injection research results

337 entries

Activation-Steering LLM Trojan

Published 11/1/2023

Analyzed 12/28/2024

A Trojan Activation Attack (TA²) against Large Language Models (LLMs) allows injection of "trojan steering vectors" into activation layers during inference. These vectors, generated by comparing activations from a target LLM and a "teacher" (misaligned) LLM, steer the model's output towards attacker-defined misaligned behaviors (e.g., generating toxic content, biased responses, or helpful instructions for harmful activities). The attack does not require retraining or modifying model weights.

Backdoor activation attack: Attack large language models using activation steering for safety-alignment

Evaluated models: Falcon 7B, GPT-3 13B, Llama 2 13B +4 more

Source: arXiv

Adversarial In-Context Hijacking

Published 11/1/2023

Analyzed 12/28/2024

A vulnerability exists in large language models (LLMs) utilizing in-context learning (ICL). Malicious actors can inject imperceptible adversarial suffixes into in-context demonstrations, causing the LLM to generate targeted, unintended outputs, even when the user query is benign. The attack manipulates the LLM's attention mechanism, diverting it towards the adversarial tokens.

Hijacking large language models via adversarial in-context learning

Evaluated models: Llama 13B, Llama 3.1 8B, Llama 3.1 8B Instruct +3 more

Source: arXiv

Autonomous Agent Jailbreak

Published 11/1/2023

Analyzed 12/29/2024

Large Language Model (LLM)-based agents, due to their multi-agent architecture and role-based interactions, are vulnerable to adversarial attacks that exploit the system's design and agent roles. Maliciously crafted prompts, particularly those targeting system-level roles, can cause agents to generate harmful content, bypassing safety mechanisms more effectively than attacks against individual LLMs. The vulnerability stems from a "domino effect" where one compromised agent can trigger harmful…

Evil geniuses: Delving into the safety of llm-based agents

Evaluated models: GPT-3.5 Turbo, GPT-4

Source: arXiv

Custom GPT Prompt Injection

Published 11/1/2023

Analyzed 12/28/2024

A prompt injection vulnerability in OpenAI's custom GPT models allows attackers to extract the system prompt and potentially leak user-uploaded files. Attackers craft malicious prompts that manipulate the LLM into revealing sensitive information, even when defensive prompts are in place. The vulnerability is exacerbated when the model includes a code interpreter.

Assessing prompt injection risks in 200+ custom gpts

Evaluated models: Not reported

Source: arXiv

Image-Based MLLM Jailbreak

Published 11/1/2023

Analyzed 12/29/2024

Multimodal Large Language Models (MLLMs) are vulnerable to a novel attack vector where query-relevant images, generated using techniques like Stable Diffusion and typography, bypass safety mechanisms and elicit unsafe responses even when the underlying LLM is safety-aligned. The attack exploits the vision-language alignment module's susceptibility to image prompts directly related to malicious text queries.

Query-relevant images jailbreak large multi-modal models

Evaluated models: Cogvlm, Idefics, InstructBLIP +11 more

Source: arXiv

Typographic VLM Jailbreak

Published 11/1/2023

Analyzed 12/29/2024

Large Vision-Language Models (VLMs) are vulnerable to jailbreaking attacks via typographically rendered visual prompts. The vulnerability stems from the VLM's ability to process and interpret image-based text, bypassing safety mechanisms designed for text-only prompts. Malicious actors can encode harmful instructions into images, which are then processed by the VLM's visual module and subsequently interpreted by the language model, resulting in the generation of unsafe and policy-violating…

Figstep: Jailbreaking large vision-language models via typographic visual prompts

Evaluated models: Cogvlm-chat-v1.1, GPT-4V, Llava-v1.5-vicuna-v1.5-13B +4 more

Source: arXiv

AutoDAN: Interpretable LLM Jailbreak

Published 10/1/2023

Analyzed 12/29/2024

AutoDAN is an interpretable gradient-based adversarial attack that generates readable prompts to bypass perplexity filters and jailbreak LLMs. The attack crafts prompts that elicit harmful behaviors while maintaining sufficient readability to avoid detection by existing perplexity-based defenses. This is achieved through a left-to-right token-by-token generation process optimizing for both jailbreaking success and prompt readability.

Autodan: Automatic and interpretable adversarial attacks on large language models

Evaluated models: GPT-3.5 Turbo, GPT-4, Guanaco 7B +4 more

Source: arXiv

Hidden Prompt Injection Attacks

Published 10/1/2023

Analyzed 12/28/2024

Large Language Models (LLMs) are vulnerable to Compositional Instruction Attacks (CIA), where malicious prompts are embedded within seemingly harmless instructions. This allows attackers to bypass safety mechanisms and elicit harmful responses from the model, even if the individual components of the prompt would be flagged as safe. The attack exploits the model's inability to correctly identify underlying malicious intent within composite instructions.

Prompt packer: Deceiving llms through compositional instruction with hidden attacks

Evaluated models: ChatGLM2 6B, GPT-3.5 Turbo, GPT-4

Source: arXiv

In-Context LLM Jailbreak

Published 10/1/2023

Analyzed 12/29/2024

Large Language Models (LLMs) are vulnerable to In-Context Attacks (ICA) and susceptible to mitigation via In-Context Defense (ICD). ICA leverages a small number of harmful demonstration examples within a prompt to elicit harmful responses from the LLM, even if it is otherwise safety-aligned. ICD counteracts ICA by prepending safe demonstration examples to the prompt, effectively reducing the likelihood of harmful output. The effectiveness of both ICA and ICD is demonstrated across multiple LLMs.

Jailbreak and guard aligned language models with only few in-context demonstrations

Evaluated models: GPT-4 0613, Llama 2 7B Chat, Mistral-7B-v2 +4 more

Source: arXiv

LLM Red Teaming Framework

Published 10/1/2023

Analyzed 12/28/2024

A vulnerability in large language models (LLMs) allows attackers to craft malicious prompts that induce the LLM to generate harmful content, such as fraudulent material, racist remarks, or instructions for illegal activities. The vulnerability arises from the LLM's inability to reliably distinguish between benign and malicious instructions disguised within seemingly innocuous prompts. Attackers can exploit this by leveraging techniques like obfuscation, code injection/payload splitting, and…

Attack prompt generation for red teaming and defending large language models

Evaluated models: Not reported

Source: arXiv

Related research categories