SOURCE-LINKED INTELLIGENCE

Tail-Replay: Escaping the Curse of Linear Attention in Prefix Caching for Hybrid LLMs

arXiv · AI, language, vision and robotics · article · Aug 31, 2026 · UTC

Hybrid large language models interleave full-attention layers with linear-attention layers to reduce the cost of long-context inference. This structure complicates prefix caching: full-attention key-value caches are token-addressable, whereas linear-attention layers maintain recurrent states that cannot be rolled back to arbitrary prefix boundaries. Existing hybrid prefix caching methods address this mismatch by storing recurrent-state checkpoints. As a result, token-level matches are directly usable only at positions aligned with stored checkpoints, constraining prefix reuse to a discrete set

Read original source ↗ Open in workspace

recordType

paper

region

Global