πref(y2|x) − β log π∗(y1|x)
πref(y1|x)

1 + exp

(cid:17)

(4)

(6)

(cid:17)

(cid:16)

The derivation is in Appendix A.2. While Eq. 6 uses the Bradley-Terry model, we can similarly
derive expressions under the more general Plackett-Luce models [30, 21], shown in Appendix A.3.
Now that we have the probability of human preference data in terms of the optimal policy rather than
the reward model, we can formulate a maximum likelihood objective for a parametrized policy πθ.
Analogous to the reward modeling approach (i.e. Eq. 2), our policy objective becomes:
πθ(yl | x)
πref(yl | x)

LDPO(πθ; πref) = −E(x,yw,yl)∼D

πθ(yw | x)
πref(yw | x)

− β log

(cid:19)(cid:21)

(cid:18)

β log

log σ

(cid:20)

(7)

.

This way, we fit an implicit reward using an alternative parameterization, whose optimal policy is
simply πθ. Moreover, since our procedure is equivalent to fitting a reparametrized Bradley-Terry

4
Direct Preference Optimization