VLM Visual-Textual Misalignment

Description: Vision-Language Models (VLMs) are vulnerable to pixel-level adversarial image perturbations. An attacker can inject $\ell_p$-bounded, human-imperceptible noise into an input image to manipulate the model's multi-modal embedding space. This reliably causes the VLM to generate incorrect textual responses, hallucinate non-existent objects, or misclassify subjects, effectively decoupling the model's reasoning from the actual visual evidence. The vulnerability is exploitable via both white-box gradient-based attacks (e.g., PGD) and black-box transfer attacks against closed-source APIs.

Examples:

• Misclassification: Applying a standard $\ell_\infty$ PGD perturbation ($\epsilon=2/255$) to an image of a t-shirt forces the VLM to confidently output "jeans".
• Targeted Hallucination: Injecting transfer-based adversarial noise into MS COCO images causes the VLM to output captions that actively hallucinate non-existent objects, misstate object counts, or confuse scene types, despite clear visual evidence to the contrary.

Impact: An attacker can deterministically subvert the integrity of VLM outputs without altering the textual prompt. When deployed in security-critical or automated environments—such as medical imaging diagnosis, autonomous driving, or robotic control—this can lead to severe misclassifications, corrupted reasoning chains, and dangerous downstream actions.

Affected Systems: General-purpose Vision-Language Models, including:

• LLaVA-1.5 (7B and 13B)
• DeepSeek-VL-1.3B
• InternVL3-2B
• Ovis2-4B
• Commercial VLM APIs susceptible to transfer attacks (e.g., GPT-4V/GPT-5, Claude, Gemini)

Mitigation Steps: Implement test-time defense mechanisms that exploit text-side redundancy and image-side denoising, such as the Paraphrase-Decomposition-Aggregation (PDA) framework:

• Paraphrase: Query the VLM using multiple semantically or logically equivalent variations of the original text prompt. Adversarial image features are highly sensitive to exact text formulations; paraphrasing forces the model to evaluate the image from shifted semantic angles.
• Decomposition: Break complex, multi-hop queries down into a sequence of atomic, verifiable sub-questions targeting isolated visual facts (e.g., object presence, color, spatial relation).
• Aggregation: Use a secondary LLM to aggregate the answers to the atomic sub-questions via agreement-aware majority voting, discarding sporadic hallucinated responses.
• Image Pre-processing: Apply lightweight image transformations, such as JPEG compression or Random Augmentation (RA), prior to inference to wash out high-frequency adversarial noise patterns.