SOURCE-LINKED INTELLIGENCE

Exact Quantile Balancing and Load-Error Injection for Mixture-of-Experts

arXiv · AI, language, vision and robotics · article · Sep 23, 2026 · UTC

Mixture-of-Experts (MoE) training requires global load balance to prevent expert under-utilization and local balance for efficient expert-parallel execution. Existing distributed Quantile Balancing (QB) uses shard-dependent or approximate global quantiles, while token-independent expert biases cannot ensure microbatch-level balance. We introduce Exact Quantile Balancing (EQB), which computes exact global-batch BF16 quantiles with negligible communication, and Load-Error Injection (LEI), which injects local load errors directly into router-score gradients. On 7.5B-parameter MoEs trained for up

Read original source ↗ Open in workspace

recordType

paper

region

Global