Black-box Vulnerabilities

Large Language Model (LLM) tool-calling systems are vulnerable to adversarial tool injection attacks. Attackers can inject malicious tools ("Manipulator Tools") into the tool platform, manipulating the LLM's tool selection and execution process. This allows for privacy theft (extracting user queries), denial-of-service (DoS) attacks against legitimate tools, and unscheduled tool-calling (forcing the use of attacker-specified tools regardless of relevance). The attack exploits vulnerabilities…

A vulnerability in LLM-based agents, dubbed AI Agent Injection (AI²), allows attackers to hijack the agent's actions by manipulating the agent's memory retrieval mechanism. The attack involves two main steps: (1) Stealing action-aware knowledge from the agent's memory using crafted adversarial queries targeting the retriever module and (2) Generating Trojan prompts consisting of a Trojan string and hijacking instructions. The Trojan string is designed to manipulate the retriever into…

Large Language Models (LLMs) employing reinforcement learning from human feedback (RLHF) for safety alignment are vulnerable to a novel "alignment-based" jailbreak attack. This attack leverages a best-of-N sampling approach with an adversarial LLM to efficiently generate prompts that bypass safety mechanisms and elicit unsafe responses from the target LLM, without requiring additional training or access to the target LLM's internal parameters. The attack exploits the inherent tension between…

Large Audio-Language Models (LALMs) are vulnerable to a stealthy adversarial jailbreak attack, AdvWave, which leverages a dual-phase optimization to overcome gradient shattering caused by audio discretization. The attack crafts adversarial audio by adding perceptually realistic environmental noise, making it difficult to detect. The attack also dynamically adapts the adversarial target based on the LALM's response patterns.

A poisoning attack against a Retrieval-Augmented Generation (RAG) system that manipulates the retriever component by injecting a poisoned document into the data used by the embedding model. This poisoned document contains modified and incorrect information. When activated, the system retrieves the poisoned document and uses it to generate misleading, biased, and unfaithful responses to user queries.

Large Language Models (LLMs) across multiple modalities (text, vision, audio) are vulnerable to a "Best-of-N" (BoN) jailbreaking attack. This attack repeatedly submits slightly modified versions of a harmful prompt (e.g., text with altered capitalization, images with modified text style, audio with altered pitch or speed) until a safety mechanism is bypassed and a harmful response is elicited. The effectiveness of the attack scales with the number of attempts (N). While individual…

A bimodal adversarial attack, PBI-Attack, can manipulate Large Vision-Language Models (LVLMs) into generating toxic or harmful content by iteratively optimizing both textual and visual inputs in a black-box setting. The attack leverages a surrogate LVLM to inject malicious features from a harmful corpus into a benign image, then iteratively refines both image and text perturbations to maximize the toxicity of the model’s output as measured by a toxicity detection model (Perspective API or…

Large Language Models (LLMs) are vulnerable to human-readable adversarial prompts crafted using situational context derived from movie scripts. These prompts, which combine a malicious prompt, a seemingly innocuous adversarial insertion, and relevant contextual information, can bypass LLMs' safety mechanisms and elicit harmful responses. The technique leverages the LLM's ability to understand context and generate responses consistent with that context to mask the malicious intent. The…

DiffusionAttacker exploits a vulnerability in Large Language Models (LLMs) allowing manipulation of prompts to elicit harmful responses, even when the model incorporates safety mechanisms. The attack leverages a sequence-to-sequence diffusion model to rewrite harmful prompts, making them appear harmless to the LLM's internal representation while preserving their original semantic meaning. This bypasses safety filters and elicits undesired outputs.

A vulnerability exists in Large Language Model (LLM)-based time series forecasting architectures, specifically affecting models such as TimeGPT, LLMTime, and TimeLLM. These models are susceptible to a gradient-free, black-box adversarial attack method termed Directional Gradient Approximation (DGA). An attacker can inject imperceptible perturbations into the historical time series input window (lookback window) to manipulate the model's output. By treating the model as a black box and…