SOURCE-LINKED INTELLIGENCE

Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs

arXiv · AI, language, vision and robotics · article · Sep 10, 2026 · UTC

Prefix caching can reduce the time to first token (TTFT) of long-context LLM requests by reusing previously computed key-value (KV) states, but for short prefixes or fast GPUs, recomputation can be faster than loading from an external cache. We characterize this tradeoff in vLLM across GPU, CPU, and NVMe tiers using synthetic workloads, long-context benchmarks, production traces, and find that cache performance depends on transfer granularity, intermediate memory use, and when transfers enter the request schedule, not only on device bandwidth. These findings motivate py-kvcache, a vLLM KV Offl

Read original source ↗ Open in workspace

recordType

paper

region

Global