SOURCE-LINKED INTELLIGENCE

Distribution-Consistent Inference for Dynamic Sparse Mixture-of-Experts

arXiv · AI, language, vision and robotics · article · Sep 8, 2026 · UTC

Mixture-of-Experts (MoE) architectures have emerged as a powerful paradigm for scaling model capacity while preserving efficient inference in large foundation models. However, most MoE models use a fixed top-$k$ expert selection policy, assigning the same expert budget to every token even when fewer experts may be sufficient. Inference-time dynamic top-$k$ routing can reduce computation without retraining, but existing methods often overlook the distributional shift caused by deviating from the training-time routing configuration. We show that reducing the number of activated experts consisten

Read original source ↗ Open in workspace

recordType

paper

region

Global