Direct preference optimization loss

Raises the reference-relative likelihood of preferred responses over dispreferred responses without fitting a separate reward model during policy optimization.

D P O loss is the negative expected log sigmoid of beta times the winner policy-to-reference log ratio minus beta times the loser policy-to-reference log ratio.

Make the preferred answer gain more probability than the rejected answer, measured against the same frozen baseline.

Negative log sigmoid is the pairwise classification loss; the inner difference decides which response the policy favors relative to the reference.

The reference policy prevents the comparison from depending only on raw sequence likelihood and makes the objective equivalent to an implicit reward-model formulation under the paper's derivation.