SOURCE-LINKED INTELLIGENCE

Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss

arXiv · AI, language, vision and robotics · article · Aug 28, 2026 · UTC

We study the scaling behavior of learning rate and batch size in pretraining dense large language models on English-prevalent corpora. Beyond scaling jointly optimal learning rates and batch sizes, we investigate their marginal evolution with model capacity and data scale and develop a model that captures these relationships. As we employ a Warmup-Stable-Decay learning rate schedule, we further investigate the gains from learning rate annealing over a broad range of hyperparameters settings, models and data budgets, and whether the optimal learning rate and batch size transfer between the stab

Read original source ↗ Open in workspace

recordType

paper

region

Global