Direct Preference Optimization: Your Language…

DPO turns language-model alignment from a multi-stage reinforcement-learning pipeline into a direct preference-learning objective. Read it to see how one mathematical reparameterization can remove the separate reward model and make preference tuning much simpler to train.

Reading focus: Why standard RLHF separates preference modeling from policy optimization. How DPO expresses the implicit reward through the language model and a reference policy. Why a classification-style loss can make preference tuning simpler and more stable than PPO-based RLHF.

NeurIPS 2023. Rafailov et al.. 40 min read, easy difficulty.