SOURCE-LINKED INTELLIGENCE

When Should a VLM Look? Paying Only for Visual Calls That Were Needed and Used

arXiv · AI, language, vision and robotics · article · Sep 19, 2026 · UTC

Vision-language agents that crop and zoom are trained with rewards that credit a successful tool call, yet a successful call does not show that the model needed to look or used the pixels it received. On our cold-start checkpoint only 10% to 12% of visual calls were both needed and used, and released agents make spurious calls 36% to 87% of the time on individual benchmarks. Outcome rewards, judge rewards, and branch probes each observe one side of this failure, and about two thirds of what an outcome reward pays goes to calls that were neither needed nor used. CounterCredit asks both question

Read original source ↗ Open in workspace

recordType

paper

region

Global