SOURCE-LINKED INTELLIGENCE

Parallelism Strategy Chaining for Fast Training Convergence

arXiv · AI, language, vision and robotics · article · Sep 7, 2026 · UTC

Selecting a parallelism strategy - the configuration of data, tensor, and pipeline parallelism degrees together with micro- and global-batch sizes - largely determines the training efficiency of large language models. State-of-the-art methods search for a parallelism strategy offline and select the single strategy that minimizes per-iteration time. But we find that they neglect the target validation perplexity and time-to-perplexity (TTP). In particular, our analysis reveals that the best strategy yielding the fastest perplexity improvement changes multiple times during training. As a result,

Read original source ↗ Open in workspace

recordType

paper

region

Global