SOURCE-LINKED INTELLIGENCE

The Undetected Damage of Quantization on Retrieval and How to Fix It

arXiv · AI, language, vision and robotics · article · Sep 21, 2026 · UTC

We show that a quantized model that keeps its classification accuracy still changes $14$ to $46\%$ of its top-1 retrieval results, and that aggregate ranking metrics reveal only part of this damage. We tie this failure to the gap between the two highest scores and use that gap to decide when a quantized answer can be trusted and where additional precision should be spent. We show that the top-1 result is guaranteed to survive quantization only when this gap exceeds twice the largest rounding error. In classification, scores are the logits, and the loss function pushes the correct class away fr

Read original source ↗ Open in workspace

recordType

paper

region

Global