SOURCE-LINKED INTELLIGENCE

Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation

arXiv · AI, language, vision and robotics · article · Sep 18, 2026 · UTC

Mixture-of-Experts (MoE) models are increasingly deployed alongside Speculative Decoding (SD) to accelerate inference, but combining the two is challenging. SD improves the inference speed of dense models by verifying groups of tokens in parallel. However, the inference speedup for SD with MoEs depends heavily on the number of tokens being verified. Using more verification tokens results in more experts being transferred from DRAM to the Neural Processing Unit (NPU), which increases the memory transfer cost. This negatively impacts model runtime, as memory transfer is typically the bottleneck

Read original source ↗ Open in workspace

recordType

paper

region

Global