Language Model Security Database

Large Language Model (LLM) based search engines utilizing Retrieval-Augmented Generation (RAG) are vulnerable to ranking manipulation attacks via indirect prompt injection. Adversaries can embed optimized adversarial triggers or crafted semantic patterns within external webpage content. When these manipulated documents are retrieved and integrated into the LLM's context window alongside a user query, the adversarial content disrupts the model's contextual understanding. This results in the LLM…

LLM-based relevance assessment frameworks, such as the Umbrela system, are vulnerable to evaluation subversion and artificial score inflation due to evaluation circularity and LLM "narcissism" (an LLM's inherent bias toward favoring LLM-generated outputs). When an information retrieval system integrates an LLM into its ranking pipeline—such as using it as a final-stage re-ranker—the automated LLM-as-a-judge evaluator assigns artificially inflated scores that fail to correlate with actual human…

A hybrid multimodal jailbreaking attack, dubbed JMLLM, exploits vulnerabilities in 13 popular large language models (LLMs) across text, image, and speech modalities. The attack leverages alternating translation, word encryption, feature collapse in images, and harmful text injection to bypass safety mechanisms and elicit harmful responses. Success rates vary across LLMs and modalities, with some models exhibiting significantly higher vulnerability than others.

Large Language Models (LLMs) are vulnerable to human-readable adversarial prompts crafted using situational context derived from movie scripts. These prompts, which combine a malicious prompt, a seemingly innocuous adversarial insertion, and relevant contextual information, can bypass LLMs' safety mechanisms and elicit harmful responses. The technique leverages the LLM's ability to understand context and generate responses consistent with that context to mask the malicious intent. The…

A vulnerability in LLM-based agents, dubbed AI Agent Injection (AI²), allows attackers to hijack the agent's actions by manipulating the agent's memory retrieval mechanism. The attack involves two main steps: (1) Stealing action-aware knowledge from the agent's memory using crafted adversarial queries targeting the retriever module and (2) Generating Trojan prompts consisting of a Trojan string and hijacking instructions. The Trojan string is designed to manipulate the retriever into…

Large Language Models (LLMs) are vulnerable to a novel prompt injection attack using universal and context-independent triggers. These triggers, once discovered for a specific LLM, allow precise control over the model's output regardless of the prompt context or desired output content, enabling adversaries to force the generation of arbitrary text. The attack utilizes a gradient-based optimization technique to discover these triggers.

Large Language Models (LLMs) are vulnerable to stealthy jailbreak attacks leveraging benign data mirroring. Attackers train a local "mirror model" on benign data obtained from the target LLM. This mirror model, mimicking the target's behavior, is then used to generate adversarial prompts, which are subsequently deployed against the target LLM, bypassing content moderation systems due to the lack of overtly malicious content in the initial data gathering phase.

Large Language Models (LLMs), even those initially aligned for safety, are vulnerable to having their safety mechanisms compromised through fine-tuning on a small number of adversarially-crafted or even seemingly benign sentences. Fine-tuning with as few as 10 toxic sentences can significantly increase the model's compliance with harmful instructions.

Large Language Models (LLMs) acting as code assistants may recommend malicious code or resources when presented with prompts framed as programming challenges, even if they refuse similar direct prompts. This occurs due to insufficient context-aware safety mechanisms. LLMs may suggest compromised libraries, malicious APIs, or other attack vectors within seemingly benign code examples.

Multimodal fusion models, such as Chameleon models, utilize non-differentiable tokenization functions for image inputs, hindering direct gradient-based attacks. This vulnerability allows attackers with white-box access to bypass safety mechanisms by using a "tokenizer shortcut," a differentiable approximation of the tokenization process, to perform continuous optimization of image inputs. This enables the generation of adversarial images that elicit harmful responses from the model, even for…