SOURCE-LINKED INTELLIGENCE

Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts

arXiv · AI, language, vision and robotics · article · Sep 1, 2026 · UTC

In current Mixture-of-Experts architectures, routing is performed based on representations dominated by structure shared across all tokens, limiting expert specialization. We show that contrasting each token against an Exponential Moving Average of the layer's hidden states, rather than routing on absolute magnitude, concentrates the routing signal onto a low-dimensional, highly separable subspace. Building on this, we propose the Contrastive Routing Mechanism (CoRM), which scores each expert by the gap between its affinity for the incoming token and its affinity for this shared reference stat

Read original source ↗ Open in workspace

recordType

paper

region

Global