SOURCE-LINKED INTELLIGENCE

Data Scout: Targeted Web Crawling for Domain-Specific Pretraining Corpora

arXiv · AI, language, vision and robotics · article · Sep 4, 2026 · UTC

The dominant approach to building domain-specific pretraining corpora is to filter large web archives such as CommonCrawl. This works well for popular domains but breaks down for specialized ones, where relevant content is sparse and often beyond the reach of popularity-driven crawlers. We present Data Scout, which inverts this: instead of filtering an archive, it directs a targeted crawl. An LLM expands a root topic into a taxonomy and thousands of search queries; the returned URLs (seeds) are grouped by subdomain and screened with a user-supplied classifier (the probe), admitting each subdom

Read original source ↗ Open in workspace

recordType

paper

region

Global