SOURCE-LINKED INTELLIGENCE

CompKV: Compensation-Aware KV Selection for Long-Context LLM Inference

arXiv · AI, language, vision and robotics · article · Sep 22, 2026 · UTC

Despite their strong performance, large language models (LLMs) are bottlenecked by KV cache memory traffic during long-context inference. Sparse attention is widely used to accelerate LLM inference by computing exact attention over a selected subset of tokens. To recover the contribution of tokens excluded from exact attention, recent methods apply coarse-grained compensation to the omitted attention tail. However, existing methods typically select tokens based on attention mass and only then compensate for the unselected tokens. This decoupled design overlooks their interaction: selection sho

Read original source ↗ Open in workspace

recordType

paper

region

Global