SOURCE-LINKED INTELLIGENCE

Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining

arXiv · AI, language, vision and robotics · article · Sep 7, 2026 · UTC

Natural Language Processing (NLP) in the climate domain requires models to process heterogeneous text sources, including scientific literature, policy disclosures, and synthetic reports. However, how to effectively combine diverse domain corpora during continued pretraining (CPT) remains underexplored. We introduce Climate-ModernBERT, a family of climate-adapted encoder models obtained through continued pretraining of ModernBERT-Base on three climate corpora: academic climate text, climate-filtered web data, and synthetic climate documents. We systematically compare joint continued pretraining

Read original source ↗ Open in workspace

recordType

paper

region

Global