SOURCE-LINKED INTELLIGENCE

Accelerating Sharded Data Parallelism at Scale with Federated Learning

arXiv · Artificial Intelligence · article · Sep 17, 2026 · UTC

The symbiotic scaling of artificial intelligence models and high-performance computing systems continually creates algorithmic challenges in their convergence. Foundation models (FMs) are a crucial example, requiring months-long training on thousands of cutting-edge GPUs. Sharded data parallelism (DP) is the dominant strategy to accelerate such computations by splitting data and models across multiple GPUs. However, it incurs prohibitive communication overhead when deployed at scale, particularly on multi-tier interconnects with heterogeneous performance. Inspired by the efficient communicatio

Read original source ↗ Open in workspace

recordType

paper

region

Global