SOURCE-LINKED INTELLIGENCE

Knowing When Not to Answer: Abstention and Refusal Reasoning in Vision--Language Models

arXiv · AI, language, vision and robotics · article · Sep 2, 2026 · UTC

Many medical conditions require diagnosis through detailed, multi-context clinical assessment rather than from visual appearance alone. Despite this, vision-language models (VLMs) are increasingly queried to interpret images in ways that touch on medical or diagnostic judgments, raising safety concerns when such inferences are unsupported. ASD diagnosis requires behavioral and developmental evidence, not static facial photographs. We audit whether VLMs abstain from this unanswerable paired-image query, and whether expressions sway non-abstaining choices. We introduce PARITY (Paired Assessment

Read original source ↗ Open in workspace

recordType

paper

region

Global