SOURCE-LINKED INTELLIGENCE

LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference

arXiv · AI, language, vision and robotics · article · Sep 2, 2026 · UTC

On-device LLM inference is attractive for privacy and responsiveness, but remains challenging on mobile and embedded devices because model weights far exceed available DRAM. Prior systems exploit activation sparsity and offload weights to SSD or flash storage, but face a fundamental systems trade-off: accurate sparse execution decisions require the latest context, whereas efficient computation-I/O overlap requires early prediction. As a result, existing designs either serialize execution or incur redundant weight fetches, extra computation, and large cache overheads. We present LeanStream, a s

Read original source ↗ Open in workspace

recordType

paper

region

Global