SOURCE-LINKED INTELLIGENCE

AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection

arXiv · AI, language, vision and robotics · article · Sep 5, 2026 · UTC

Aligning large language models with human preferences remains a challenge, primarily due to the critical role of preference data quality in effective alignment. Existing datasets are frequently plagued by inherent noise and distribution shifts, which inherently limit model performance. To bridge this gap, we propose AlignDiff, a preference data filtering framework driven by intrinsic model signals. AlignDiff first identifies samples with clear preferences using both positive and inverse signals, then prioritizes the more challenging samples based on the average negative log-likelihood gap, enc

Read original source ↗ Open in workspace

recordType

paper

region

Global