SOURCE-LINKED INTELLIGENCE

Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving

arXiv · AI, language, vision and robotics · article · Sep 4, 2026 · UTC

Prefix caching, in which a serving engine reuses the key and value tensors of a shared prompt prefix across requests, is enabled by default in the major open-source stacks and treated as a transparent optimization. We measure what it costs in reproducibility, and find that the cost rises sharply with weight quantization. Holding the model, decoding parameters, seed, and request order fixed, and issuing every request serially at batch size one, we ran an eighty-episode multi-turn agentic tool-use workload with caching enabled and disabled across two engines and four weight formats. Enabling the

Read original source ↗ Open in workspace

recordType

paper

region

Global