Architecture Components

Multimodal Research

Security research involving multimodal AI systems

42 matching entries out of 176 in this category

Multimodal research results

42 entries

Selective refusal gaps in visual question answering

Published Sep 4, 2026Analyzed Sep 9, 2026Source: arXiv

KoNA measures whether vision-language models answer valid image questions while refusing unsafe components or correcting unsupported premises. Its 9,300 question-answer pairs include mixed and fully answerable controls.

Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models

Evaluated models: InternVL3 2B Instruct, InternVL3-78B-Instruct, Qwen 2.5 VL 3B Instruct+5 more

InternVL3 2B Instruct
InternVL3-78B-Instruct
Qwen 2.5 VL 3B Instruct
Qwen 2.5 VL 72B Instruct
GPT-5
Gemini 2.5 Flash
InternVL3-2B-KoNA
Qwen2.5-VL-3B-KoNA

PII leakage in document-image redaction benchmarks

Published Sep 2, 2026Analyzed Sep 9, 2026Source: arXiv

Document-image PII detection can miss identifiers despite improving average localization scores. LeakageBench evaluates 500 pages with 11,954 annotations.

LeakageBench: Document-Level Leakage Risk for Redacting Personally Identifiable Information in Document Images

Evaluated models: GLiNER-base, GLiNER-multi-PII, NVIDIA GLiNER PII+5 more

GLiNER-base
GLiNER-multi-PII
NVIDIA GLiNER PII
GLiNER2
Qwen3-VL-32B
InternVL3 38B
GPT-5.4
GPT-5.5

Multilingual Flowchart Jailbreaks in Vision-Language Models

Published Jun 5, 2026Analyzed Jul 20, 2026Source: arXiv

MLingualFC is a reproducible black-box safety evaluation showing that harmful instructions rendered as multilingual flowchart images can bypass vision-language model safeguards more often than equivalent text-only inputs. The paper evaluates horizontal, vertical, and tortuous layouts across English, Hindi, Punjabi, Spanish, Romanian, and German. Reported results vary substantially by language, script, layout, and model; these are paper-reported measurements, not independently verified…

MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models

Evaluated models: Qwen 2.5 VL 3B Instruct, Gemma-4-E4B-it, Pangea-7B

Query-Agnostic Poisoning of Medical Multimodal RAG

Published May 11, 2026Analyzed Jul 20, 2026Source: arXiv

M3Att demonstrates a reproducible knowledge-poisoning issue in medical multimodal RAG: an attacker with limited corpus-distribution knowledge can insert paired image-text entries whose visually perturbed images are broadly retrieved and whose clinically plausible misinformation steers downstream generation. The paper evaluates both white-box and black-box retrieval optimization and reports degraded diagnostic and report-generation utility across multiple datasets, retrievers, and LVLMs. These…

Knowledge Poisoning Attacks on Medical Multi-Modal Retrieval-Augmented Generation

Evaluated models: GPT-4o, GPT-5 Chat, Gemini 2.5 Flash+5 more

GPT-4o
GPT-5 Chat
Gemini 2.5 Flash
Claude Haiku 4.5
LLaVA Med
CLIP ViT-Large-Patch14-336
BGE-VL-base
SigLIP-SO400M-Patch14-384

Diverse VLA Linguistic Fragility

Published Apr 7, 2026Analyzed Apr 10, 2026Source: arXiv

Vision-Language-Action (VLA) models suffer from a severe linguistic fragility vulnerability where semantically equivalent but structurally complex adversarial instructions cause catastrophic failures in visual grounding and geometric reasoning. Attackers can reliably induce physical execution failures in robotic manipulation tasks by applying semantic-preserving linguistic variations, such as synonymous rephrasing, syntactic restructuring, or the addition of fine-grained compositional…

Uncovering Linguistic Fragility in Vision-Language-Action Models via Diversity-Aware Red Teaming

Evaluated models: Pi-Zero, OpenVLA 7B, 3D-Diffuser Actor

Physical Infrared Semantic Disruption

Published Apr 3, 2026Analyzed Apr 11, 2026Source: arXiv

A vulnerability in Infrared Vision-Language Models (IR-VLMs) allows attackers to systematically degrade open-ended semantic understanding—compromising classification, captioning, and Visual Question Answering (VQA)—via a physically deployable Universal Curved-Grid Patch (UCGP). Instead of manipulating explicit text labels, the attack disrupts the clean-category manifold in the model's visual representation space by maximizing orthogonal deviation energy from the principal subspace and forcing…

Revealing Physical-World Semantic Vulnerabilities: Universal Adversarial Patches for Infrared Vision-Language Models

Evaluated models: InstructBLIP

VLM Visual-Textual Misalignment

Published Apr 1, 2026Analyzed Apr 11, 2026Source: arXiv

Vision-Language Models (VLMs) are vulnerable to pixel-level adversarial image perturbations. An attacker can inject $\ell_p$-bounded, human-imperceptible noise into an input image to manipulate the model's multi-modal embedding space. This reliably causes the VLM to generate incorrect textual responses, hallucinate non-existent objects, or misclassify subjects, effectively decoupling the model's reasoning from the actual visual evidence. The vulnerability is exploitable via both white-box…

PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks

Evaluated models: LLaVA 1.5 7B, LLaVA 1.5 13B, DeepSeek VL 1.3B+2 more

LLaVA 1.5 7B
LLaVA 1.5 13B
DeepSeek VL 1.3B
InternVL3 2B
Ovis2 4B

X-Shaped Sparse VLM Attack

Published Mar 30, 2026Analyzed Apr 11, 2026Source: arXiv

A vulnerability in Vision-Language Models (VLMs) relying on shared visual-textual representation spaces allows attackers to induce transferable cross-task semantic failures using an X-shaped Sparse Pixel Attack (XSPA). Attackers craft imperceptible adversarial perturbations restricted to a fixed geometric prior—two intersecting diagonal lines comprising approximately 1.76% of the image pixels. By jointly optimizing a classification objective with cross-task semantic guidance (target-semantic…

XSPA: Crafting Imperceptible X-Shaped Sparse Adversarial Perturbations for Transferable Attacks on VLMs

Evaluated models: InstructBLIP

Agentic Robot Instruction Attack

Published Mar 26, 2026Analyzed Apr 10, 2026Source: arXiv

Vision-Language-Action (VLA) models are vulnerable to targeted, low-budget textual perturbations in their natural-language instruction inputs, which can maliciously alter sequential decision-making and downstream physical robotic behavior. Because VLA policies tightly couple language, perception, and control, bounded edits—such as character-level typos, token attribute swaps, or prompt-level uncertainty clauses—propagate through the model's execution trajectory. This allows a black-box…

SABER: A Stealthy Agentic Black-Box Attack Framework for Vision-Language-Action Models

Evaluated models: Not reported

VLM E-commerce Attack Surface

Published Mar 17, 2026Analyzed Apr 11, 2026Source: arXiv

LLaVA-v1.5-7B, when deployed as a vision-language autonomous agent, is highly vulnerable to adversarial image perturbations. An attacker can inject imperceptibly modified images into a web environment (such as an e-commerce storefront). When the VLM agent captures a screenshot containing the perturbed image, the visual noise forces the model to misclassify the scene and output incorrect, structured JSON actions. This allows an attacker to hijack the agent's task execution, bypassing the user's…

Adversarial attacks against Modern Vision-Language Models

Evaluated models: Qwen 2.5 VL 7B Instruct, LLaVA 1.5 7B

Related research categories