SOURCE-LINKED INTELLIGENCE

PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization

arXiv · AI, language, vision and robotics · article · Aug 31, 2026 · UTC

Direct Preference Optimization (DPO) simplifies alignment through pairwise comparisons but assumes all observed preferences are reliable. Real data often violates this assumption, leading to reversed, weak, or ambiguous labels that cause harmful policy updates. To address this, we propose Posterior Label Correction DPO (PLC-DPO) to robustly optimize preferences by routing each pair's training signal as a clean, flip, or tie case. The key idea is to use the calibrated policy-reference margin as online evidence to take appropriate correction actions. This reframes noisy preference learning as ac

Read original source ↗ Open in workspace

recordType

paper

region

Global