SOURCE-LINKED INTELLIGENCE

Interface-Aware KV Cache Quantization for Dense On-Chip NVM in Long-Context LLM Decoding

arXiv · AI, language, vision and robotics · article · Sep 4, 2026 · UTC

The key-value (KV) cache is the dominant memory bottleneck in long-context large language model (LLM) decoding: every step reads it entirely, so decoding is memory-bandwidth bound. Holding a quantized KV cache in dense on-chip non-volatile memory (NVM) removes the off-chip transfer. Existing KV quantization methods, however, were designed for GPU-style memory systems: KIVI attaches per-group metadata, adding about 25% to the stored KV cache; KVQuant keeps sparse full-precision outliers that a dense array cannot hold in place. This paper examines what these structures cost when the KV cache res

Read original source ↗ Open in workspace

recordType

paper

region

Global