Impact

Integrity Research

Security research involving model-output integrity and reliability

539 matching entries

Integrity research results

539 entries

Self-Fooling LLM Prompt Attack

Published 10/1/2023

Analyzed 1/26/2025

A prompt-based adversarial attack, termed PromptAttack, can cause Large Language Models (LLMs) to generate incorrect outputs by manipulating the input prompt. PromptAttack crafts prompts that include the original input, an attack objective (to generate semantically similar but misclassified output), and attack guidance with instructions for character, word, or sentence-level perturbations. This allows an attacker to manipulate an LLM's response without direct access to its internal parameters…

An LLM can Fool Itself: A Prompt-Based Adversarial Attack

Evaluated models: GPT-3.5 Turbo

Source: arXiv

Visual Adversarial Tool Abuse

Published 10/1/2023

Analyzed 12/28/2024

A vulnerability exists in multimodal Large Language Models (LLMs) integrated with external tools. Adversarial images, visually indistinguishable from benign images, can manipulate the LLM to execute unintended tool commands, compromising the confidentiality and integrity of user resources. The attack is effective across diverse prompts, remaining stealthy both in the image itself and in the generated text response.

Misusing tools in large language models with visual adversarial examples

Evaluated models: Not reported

Source: arXiv

Auto-Generated LLM Jailbreaks

Published 9/1/2023

Analyzed 12/28/2024

Large Language Models (LLMs) are susceptible to automated jailbreak attacks using a fuzzing framework that generates variations of existing jailbreak prompts. This vulnerability allows bypassing built-in safety mechanisms, leading to the generation of harmful or unintended outputs. The vulnerability stems from the LLMs' inability to consistently recognize and reject semantically similar, but subtly different prompt variations generated through automated mutation techniques.

Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts

Evaluated models: Not reported

Source: arXiv

Chain-of-Utterance Jailbreak

Published 8/1/2023

Analyzed 12/28/2024

Large Language Models (LLMs) are vulnerable to a "Chain of Utterances" (CoU) based prompt injection attack. This attack exploits the LLM's ability to engage in multi-turn conversations and role-playing, tricking it into providing harmful or unsafe responses even when presented with safety guidelines. The attack leverages a crafted conversation between two agents ("Red-LM," a malicious agent, and "Base-LM," a seemingly helpful agent) to elicit unethical responses from the Base-LM by subtly…

Red-teaming large language models using chain of utterances for safety-alignment

Evaluated models: Not reported

Source: arXiv

Prompt-SQL Injection Vulnerability

Published 8/1/2023

Analyzed 1/26/2025

Large Language Model (LLM)-integrated web applications using Langchain (and potentially similar middleware) are vulnerable to Prompt-to-SQL (P2SQL) injection attacks. Unsanitized user prompts can be crafted to cause the LLM to generate malicious SQL queries, leading to unauthorized database access (read and write operations). This vulnerability bypasses attempts to restrict the LLM through prompt engineering alone.

From prompt injections to sql injection attacks: How protected is your llm-integrated web application?

Evaluated models: GPT-3.5 Turbo, GPT-4, PaLM 2 +1 more

Source: arXiv

Automated LLM Jailbreak Framework

Published 7/1/2023

Analyzed 12/28/2024

The MASTER KEY framework exploits timing-based characteristics of Large Language Model (LLM) chatbot responses to infer internal defense mechanisms and automatically generate jailbreak prompts. This allows bypassing safety restrictions and eliciting responses violating usage policies, including generation of illegal, harmful, privacy-violating, and adult content. The framework utilizes a three-step process: reverse-engineering defenses via time-based analysis, creating proof-of-concept…

MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots

Evaluated models: ERNIE, GPT-3.5 Turbo, GPT-4

Source: arXiv

Universal Adversarial LLM Jailbreak

Published 7/1/2023

Analyzed 12/28/2024

Aligned large language models (LLMs) are vulnerable to a universal and transferable adversarial suffix attack. Appending a specific, automatically generated suffix to a wide range of prompts, even those requesting objectionable content, causes the models to generate harmful or objectionable responses instead of refusing the request. The attack's success rate is significantly higher on GPT-based models.

Universal and transferable adversarial attacks on aligned language models

Evaluated models: ChatGLM 6B, Claude Instant 1, Claude 2 +12 more

Source: arXiv

HouYi Prompt Injection

Published 6/1/2023

Analyzed 12/29/2024

A prompt injection vulnerability allows attackers to manipulate the behavior of Large Language Model (LLM)-integrated applications by crafting malicious prompts that override the application's intended functionality. Attackers can achieve this by constructing prompts that cause the LLM to interpret malicious payloads as instructions, rather than data, leading to unintended actions such as data leakage, unauthorized LLM usage, or application mimicry. This vulnerability exploits the way user…

Prompt Injection attack against LLM-integrated Applications

Evaluated models: GPT-3.5

Source: arXiv

Prompt Engineering Jailbreak

Published 5/1/2023

Analyzed 12/29/2024

Large Language Models (LLMs), specifically ChatGPT versions 3.5 and 4.0, are vulnerable to prompt engineering attacks that circumvent built-in content restrictions. Attackers can craft malicious prompts, categorized into "pretending," "attention shifting," and "privilege escalation" techniques, to elicit responses containing prohibited content (e.g., instructions for illegal activities, generation of harmful content). The vulnerability stems from the LLM's inability to reliably distinguish…

Jailbreaking chatgpt via prompt engineering: An empirical study

Evaluated models: GPT-3.5 Turbo, GPT-4

Source: arXiv

Related research categories