− V ∗(xT , c)

T(cid:88)

t=0

Using a telescoping sum through the diffusion chain we are left with

since by definition all intermediate rewards are zero. If we assume both diffusion chains start from the same state and plug
this result into the preference formulation of Eq. 3 we obtain the objective of Eq. 11. Here we optimize the same objective
as prior works [6, 11], but instead of a policy gradient approach we derive our objective as an off-policy learning problem in
the same MDP. This not only simplifies the algorithm significantly, but justifies our sampling choices in Eq. 13 and we do
not have to sample through the entire difussion chain.
DiffusionModelAlignmentUsing Direct Preference Optimization