SOURCE-LINKED INTELLIGENCE

Fine PT-PT Web: A High-Quality 41 Billion Tokens Data Collection of the European Portuguese Web

arXiv · AI, language, vision and robotics · article · Sep 7, 2026 · UTC

Curating Web corpora for regional language variants like European Portuguese (PT-PT) is heavily bottlenecked by dialectal overlap (mainly with PT-BR) and data processing scale. This paper presents an efficient pipeline to curate a production-ready PT-PT corpus from the Portuguese Web, spanning 411 TB of raw data from Arquivo.pt. We introduce a novel post-scraping block that removes boilerplate and line duplicates prior to filtering. This early-stage intervention increases final document yield by 19.04% by rescuing valid text that standard heuristic filters prematurely discard. Integrated with

Read original source ↗ Open in workspace

recordType

paper

region

Global