Skip to main content
LLM Security Database
Skip to research details
Back to research findings
LMVD-ID: 41ff4d29
Paper published September 4, 2026
Entry analyzed September 9, 2026
Paper-reported evidence
Confidence: Source-linked

The LMVD-ID is an internal research identifier, not an official CVE identifier.

Selective refusal gaps in visual question answering

KoNA measures whether vision-language models answer valid image questions while refusing unsafe components or correcting unsupported premises. Its 9,300 question-answer pairs include mixed and fully answerable controls.

Model layer
Vision
Multimodal
Safety
Reliability
Hallucination
Fine-tuning
Read primary paperBibTeX citation

Paper-evaluated models(8)

InternVL3 2B Instruct, InternVL3-78B-Instruct, Qwen 2.5 VL 3B Instruct +5 more
  • InternVL3 2B Instruct
  • InternVL3-78B-Instruct
  • Qwen 2.5 VL 3B Instruct
  • Qwen 2.5 VL 72B Instruct
  • GPT-5
  • Gemini 2.5 Flash
  • InternVL3-2B-KoNA
  • Qwen2.5-VL-3B-KoNA

Description

KoNA measures whether vision-language models answer valid image questions while refusing unsafe components or correcting unsupported premises. Its 9,300 question-answer pairs include mixed and fully answerable controls.

Examples

See the primary benchmark (opens in a new tab).

Impact

Default compound-query Safety accuracy is 0.01 for Qwen2.5-VL-3B and 0.91 for GPT-5. KoNA tuning raises the small Qwen model to 0.98, while fully answerable accuracy moves from 0.73 to 0.71. Tuning covers two small open models; this is a selective-response benchmark, not adaptive jailbreak testing or a production failure estimate.

Affected Systems

  • The six evaluated vision-language models and two KoNA-tuned variants; GPT-5-mini serves as an evaluator.

Mitigation Steps

  • Score valid and unsafe components separately.
  • Evaluate answerability alongside refusal behavior after tuning.
  • Use human sampling to audit automatic safety and factuality judgments.

Evidence

Research context and confidence

Evidence and verification
Paper-reported; independent reproduction is not documented.
Primary source plus a dedicated evidence section.
Severity
Not rated by this catalog.
Source and publication type
arXiv · Research preprint.
Peer-review status is not provided by this source.
Author and publication status
Author metadata is not stored; see the primary paper.
Threat model and attacker access
Not explicitly classified; consult the primary paper..
Related deployment categories
Vision-language models; Fine-tuning
Taxonomy labels only; paper-specific deployment prerequisites are not inferred.
Affected systems
The six evaluated vision-language models and two KoNA-tuned variants; GPT-5-mini serves as an evaluator.

Research Paper

Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models

Primary source: arXiv. Findings are reported by the cited research and have not been independently verified.

View Paper

Related research