SOURCE-LINKED INTELLIGENCE

TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching

arXiv · AI, language, vision and robotics · article · Sep 18, 2026 · UTC

Large language models (LLMs) are moving onto mobile devices for increasingly diverse workloads over text, images, video, and audio. These applications often require long contexts, making the Key-Value (KV) cache a dominant memory bottleneck because it grows linearly with sequence length and is accessed at every decoding step. Prior work reduces KV-cache footprint through low-rank compression, token eviction, or flash offloading, but the resulting reconstruction overhead, irreversible token loss, or I/O stalls can offset the benefit of saving memory. We present TierKV, a mobile LLM inference fr

Read original source ↗ Open in workspace

recordType

paper

region

Global