SOURCE-LINKED INTELLIGENCE

mzCache: On-Device LLM Memory Management under Multitasking

arXiv · AI, language, vision and robotics · article · Sep 1, 2026 · UTC

On-device mobile Large Language Model (LLM) inference is gaining significant attention. However, mobile devices operate in highly dynamic multitasking environments where users frequently switch between applications. This creates memory pressure, forcing LLM memory (model weights and KV cache) to be evicted by the operating system. When a new inference request arrives, the inference system must restore the evicted memory through slow storage reads or recompute the entire KV cache, severely degrading responsiveness. To address this, we present mzCache, an on-device LLM inference system with spec

Read original source ↗ Open in workspace

recordType

paper

region

Global