SOURCE-LINKED INTELLIGENCE

BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inference

arXiv · AI, language, vision and robotics · article · Sep 11, 2026 · UTC

GPU memory bandwidth and capacity limit throughput in large language model (LLM) inference. The GPU memory system consists of a primary tier of high-bandwidth memory (HBM) and a secondary tier of host memory connected via CPU-to-GPU interconnect. Current serving systems treat the tiers hierarchically: they serve exclusively from HBM when data fits, and otherwise prefetch data from host memory to HBM before use. In both cases, the host memory bandwidth is never well utilized. Prefetching expands capacity by utilizing host memory, but consumes HBM bandwidth for writes, reducing the bandwidth ava

Read original source ↗ Open in workspace

recordType

paper

region

Global