SOURCE-LINKED INTELLIGENCE

Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting

arXiv · AI, language, vision and robotics · article · Sep 20, 2026 · UTC

Large language models (LLMs) drift out of date the moment their pretraining ends, yet retraining from scratch is prohibitively expensive. Continued pretraining (CPT) is the natural remedy, but it is typically evaluated through a continual learning lens that assumes disjoint data streams. This is a poor fit for time-incremental updates on web-scale crawls, where successive snapshots share substantial URL overlap by design. We study time-incremental CPT in this realistic regime: continued pretraining on FineWeb-Edu dumps drawn strictly from after each model's knowledge cutoff, evaluated across s

Read original source ↗ Open in workspace

recordType

paper

region

Global