4

model, it enjoys certain theoretical properties, such as consistencies under suitable assumption of the
preference data distribution [4]. In Section 5, we further discuss theoretical properties of DPO in
relation to other works.
What does the DPO update do? For a mechanistic understanding of DPO, it is useful to analyze the
gradient of the loss function LDPO. The gradient with respect to the parameters θ can be written as:

∇θLDPO(πθ; πref) =
− βE(x,yw,yl)∼D

(cid:20)

(cid:124)

σ(ˆrθ(x, yl) − ˆrθ(x, yw))

∇θ log π(yw | x)

− ∇θ log π(yl | x)

(cid:123)(cid:122)

(cid:125)

(cid:124)

(cid:123)(cid:122)

higher weight when reward estimate is wrong

increase likelihood of yw

decrease likelihood of yl

(cid:123)(cid:122)

(cid:20)

(cid:124)

(cid:125)

(cid:21)(cid:21)

,

(cid:125)
Direct Preference Optimization