SOURCE-LINKED INTELLIGENCE

A Spectral Theory of Grokking: Weight Decay induces Feature Learning

arXiv · AI, language, vision and robotics · article · Sep 22, 2026 · UTC

In grokking an early fit to the training data separates from a much later improvement in generalization. During this delay, training can move from a fixed neural tangent kernel (NTK) regime to one in which task-relevant kernel eigendirections continue to evolve. We provide a quantitative theory for how this transition from lazy to rich learning can produce delayed generalization. For homogeneous networks trained with squared loss and $L_2$ weight decay, we show that a finite residual remains after memorization, with larger residual fractions in target components associated with smaller NTK eig

Read original source ↗ Open in workspace

recordType

paper

region

Global