SOURCE-LINKED INTELLIGENCE

Pre-carved Niches: The Formation Dynamics of Modular Task Partitions in Early LLM Training

arXiv · AI, language, vision and robotics · article · Sep 1, 2026 · UTC

Large language models exhibit a modular internal organization that mirrors well-studied functional networks of the human brain, but how this organization forms during training is unknown: prior work has characterized finished models, not the formation process. We track formation step by step: we train a Pythia-410M model from scratch (two trajectories, bf16 and fp32) and run attribution patching at every step, alongside probes for gradient norms, effective updates, weight norms, and first-order loss decomposition across 14 tasks in four cognitive domains. Three findings. First, the modular map

Read original source ↗ Open in workspace

recordType

paper

region

Global