SOURCE-LINKED INTELLIGENCE

KItCAT: Knowledge Injection via Input Corruption for Auto-regressive Training

arXiv · AI, language, vision and robotics · article · Aug 31, 2026 · UTC

LLMs acquire vast amounts of knowledge during pre-training, but often lack the specialized knowledge needed to answer questions from niche sources such as manuals or technical documents unseen during pre-training. Continued pre-training (CPT) is widely used to inject such knowledge into model parameters. However, niche documents seldom repeat facts, making it difficult for CPT to robustly acquire such knowledge. Recent works address this by generating multiple paraphrases of the new knowledge, but paraphrasing is computationally expensive and typically requires powerful LLMs. In this work, we

Read original source ↗ Open in workspace

recordType

paper

region

Global