---
title: "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"
description: "DPO turns language-model alignment from a multi-stage reinforcement-learning pipeline into a direct preference-learning objective. Read it to see how one mathematical reparameterization can remove the separate reward model and make preference tuning much simpler to train."
canonical_url: "https://fanout.sh/daily/2026-07-22-direct-preference-optimization"
md_url: "https://fanout.sh/daily/2026-07-22-direct-preference-optimization.md"
last_updated: "2026-07-22"
access: "public"
---

# Direct Preference Optimization: Your Language Model is Secretly a Reward Model

DPO turns language-model alignment from a multi-stage reinforcement-learning pipeline into a direct preference-learning objective. Read it to see how one mathematical reparameterization can remove the separate reward model and make preference tuning much simpler to train.

## Paper details

- Authors: Rafailov et al.

- Venue: NeurIPS 2023

- Track: AI Research

- Difficulty: Easy

- Reading time: 40 min

- Original paper: https://arxiv.org/abs/2305.18290

## What you will learn

- Why standard RLHF separates preference modeling from policy optimization.

- How DPO expresses the implicit reward through the language model and a reference policy.

- Why a classification-style loss can make preference tuning simpler and more stable than PPO-based RLHF.

## Continue exploring

[Explore DPO](https://fanout.sh/knowledge-graph?node=ai-research%3Aconcept%3Adpo): Open Fanout's DPO concept node and connect the paper to preference data, reward models, RLHF, and post-training alignment.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md