SOURCE-LINKED INTELLIGENCE

Beyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustment

arXiv · AI, language, vision and robotics · article · Sep 5, 2026 · UTC

Retraining-free MoE compression reduces deployment memory by pruning or merging experts, but often treats the compressed checkpoint as the final artifact. We argue that this view is incomplete: compressed MoE checkpoints are better understood as compressed initializations that benefit from a tiny post-compression adjustment stage. Across two MoE LLM backbones, four pruning/merging methods, three expert-retention ratios, and 28 benchmarks, we compare LM fine-tuning and teacher-based KD under matched small-data budgets and measured GPU costs. Using only 3,000 C4 examples and a single epoch of ad

Read original source ↗ Open in workspace

recordType

paper

region

Global