Architecture Components

Vision Research

Security issues in vision-language models

128 matching entries

Vision research results

128 entries

LVLM Vision-Triggered Resource Exhaustion

Published 7/1/2025

Analyzed 7/28/2025

A resource consumption vulnerability exists in multiple Large Vision-Language Models (LVLMs). An attacker can craft a subtle, imperceptible adversarial perturbation and apply it to an input image. When this image is processed by an LVLM, even with a benign text prompt, it forces the model into an unbounded generation loop. The attack, named RECALLED, uses a gradient-based optimization process to create a visual perturbation that steers the model's text generation towards a predefined…

Resource Consumption Red-Teaming for Large Vision-Language Models

Evaluated models: LLaVA 1.5 7B, LLaVA 1.5 13B, Qwen 2.5 VL 3B Instruct +4 more

Source: arXiv

Visual Jailbreak via Context Injection

Published 7/1/2025

Analyzed 7/14/2025

Multimodal Large Language Models (MLLMs) are vulnerable to visual contextual attacks, where carefully crafted images and accompanying text prompts can bypass safety mechanisms and elicit harmful responses. The vulnerability stems from the MLLM's ability to integrate visual and textual context to generate outputs, allowing attackers to create realistic scenarios that subvert safety filters. Specifically, the attack leverages image-driven context injection to construct deceptive multi-turn…

Visual Contextual Attack: Jailbreaking MLLMs with Image-Driven Context Injection

Evaluated models: Gemini 2.0 Flash, GPT-4o, GPT-4o Mini +3 more

Source: arXiv

Benign LLM Secondary Risks

Published 6/1/2025

Analyzed 2/21/2026

Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) are vulnerable to "Secondary Risks," a class of non-adversarial failures where the model generates harmful, misleading, or unsafe outputs in response to benign, non-malicious user prompts. Unlike jailbreaks which require adversarial inputs, secondary risks arise from imperfect generalization and alignment failures during standard interactions. This vulnerability manifests primarily in two primitives: 1. Excessive…

Exploring the Secondary Risks of Large Language Models

Evaluated models: GPT-4o, Claude 3.7 Sonnet, GPT-4 Turbo +9 more

Source: arXiv

JailFlip Implicit Harm

Published 6/1/2025

Analyzed 12/8/2025

A vulnerability exists in the safety alignment mechanisms of Large Language Models (LLMs) (including GPT-4, Claude 3, Gemini, and Qwen families) leading to "Implicit Harm." Unlike traditional jailbreaks that use overtly harmful queries, this vulnerability allows remote attackers to coerce the model into providing factually incorrect, plausible, and dangerous responses to benign-looking inputs. By employing "JailFlip" techniques—specifically constructed affirmative-type or denial-type queries…

Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failures

Evaluated models: GPT-4.1, GPT-4.1 Mini, GPT-4o +3 more

Source: arXiv

Code-Mixed Phonetic Attack

Published 5/1/2025

Analyzed 9/7/2025

A vulnerability exists in multiple large language and multimodal models that allows for the bypass of safety filters through the use of code-mixed prompts with phonetic perturbations. An attacker can craft a prompt in a code-mixed language (e.g., Hinglish) and apply phonetic misspellings to sensitive keywords (e.g., spelling "hate" as "haet"). This technique causes the model's tokenizer to parse the sensitive word into benign sub-tokens, preventing safety mechanisms from flagging the harmful…

" Haet Bhasha aur Diskrimineshun": Phonetic Perturbations in Code-Mixed Hinglish to Red-Team LLMs

Evaluated models: Gemma 1.1 7B IT, GPT-4o, GPT-4o Mini +2 more

Source: arXiv

Dynamic Prompt Jailbreak

Published 5/1/2025

Analyzed 5/31/2025

GhostPrompt demonstrates a vulnerability in multimodal safety filters used with text-to-image generative models. The vulnerability allows attackers to bypass these filters by using a dynamic prompt optimization framework that iteratively generates adversarial prompts designed to evade both text-based and image-based safety checks while preserving the original, harmful intent of the prompt. This bypass is achieved through a combination of semantically aligned prompt rewriting and the injection…

GhostPrompt: Jailbreaking Text-to-image Generative Models based on Dynamic Optimization

Evaluated models: DALL-E 3, DeepSeek V3, Flux Schnell +5 more

Source: arXiv

Hidden Image Jailbreak

Published 5/1/2025

Analyzed 5/31/2025

Multimodal large language models (MLLMs) are vulnerable to implicit jailbreak attacks that leverage least significant bit (LSB) steganography to conceal malicious instructions within images. These instructions are coupled with seemingly benign image-related text prompts, causing the MLLM to execute the hidden malicious instructions. The attack bypasses existing safety mechanisms by exploiting cross-modal reasoning capabilities.

Implicit Jailbreak Attacks via Cross-Modal Information Concealment on Vision-Language Models

Evaluated models: Gemini 1.5 Pro, Gemini 2.5 Pro, GPT-4.5 +3 more

Source: arXiv

Hybrid Agent Prompt Injection

Published 5/1/2025

Analyzed 12/9/2025

Computer-Use Agents (CUAs) powered by Large Language Models (LLMs) operating in hybrid Web-OS environments are vulnerable to indirect prompt injection. Attackers can embed malicious natural language or code instructions within legitimate web content (e.g., social media forums, chat applications, shared cloud documents) that the agent processes during benign task execution. Due to the agent's inability to distinguish between trusted user instructions and untrusted environmental data, the CUA…

RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environments

Evaluated models: Claude 3.5 Sonnet, Claude 3.7 Sonnet, GPT-4o

Source: arXiv

On-Device LLM Hijacking

Published 5/1/2025

Analyzed 12/9/2025

Mobile LLM agents utilizing vision-based screen perception (OCR or Multimodal Large Language Models) are vulnerable to Visual Prompt Injection via malicious GUI overlays. An attacker holding the SYSTEM_ALERT_WINDOW permission can deploy non-focusable floating windows (using FLAG_NOT_FOCUSABLE) containing adversarial text or fabricated UI elements over legitimate applications. Because the agent captures the entire screen buffer to interpret the device state, it ingests the adversarial overlay…

From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agents

Evaluated models: GPT-4o

Source: arXiv

Pretraining Modality Gap Jailbreak

Published 5/1/2025

Analyzed 12/30/2025

Large Vision-Language Models (LVLMs) that utilize a projection layer (adapter) to bridge a vision encoder and a Large Language Model (LLM) contain a vulnerability stemming from the "Modality Gap"—a distributional distance between image and text token embeddings. This gap allows the visual modality to bypass the safety alignment (RLHF/instruction tuning) of the backbone LLM. Attackers can trigger harmful, toxic, or illegal responses to queries that would be refused in text-only contexts by…

Bootstrapping LLM Robustness for VLM Safety via Reducing the Pretraining Modality Gap

Evaluated models: LLaVA 7B, Vicuna 7B

Source: arXiv

Related research categories