SOURCE-LINKED INTELLIGENCE

MILO: Efficient Many-shot In-Context Learning with Block-wise Low-rank Compression

arXiv · AI, language, vision and robotics · article · Sep 24, 2026 · UTC

Many-shot in-context learning (ICL) enables large language models (LLMs) to adapt to complex tasks by conditioning on thousands of demonstration examples, but this paradigm shifts the inference efficiency bottleneck to the key-value (KV) cache memory. Due to the linear scaling behavior of the KV cache, storing these intermediate tensors has become a paramount challenge for both online serving and on-device deployment. To address this issue, we propose a novel compression framework, termed MILO, that exploits the low-rank redundancy inherent in many-shot contexts. Specifically, MILO features a

Read original source ↗ Open in workspace

recordType

paper

region

Global