SE(l)
ref

zte

(cid:19)(cid:19)

=

(41)

ψ = ||ϵqt − ψ(c, t, qt(xd

0))||2

Here we use SE(d)

2 to denote the L2 error in the noise prediction of model ψ operating on
the noisy qt(xd
ref is the model of
the reference policy pref. Note that these SE terms are the standard diffusion training objective from Eq. (2). Continuing to
simplify the above yields:

0) with corresponding conditioning (c, t) (d ∈ {w, l}). Here the model associated with SE∗

(32)

(cid:19)(cid:21)

(34)

(35)

(36)

(37)

(38)

(39)

(40)

(42)

− log σ

log

β
(cid:18)

zte

zte

(cid:16)

− b2
t
2a2
t σ2
t

− b2
t
2a2
t σ2
t

SE(w)

θ

SE(w)

ref

− log

zte

zte

− b2
t
2a2
t σ2
t

− b2
t
2a2
t σ2
t

SE(l)

θ

SE(l)
ref

= − log σ

−β

b2
t
2a2
t σ2
t

(SE(w)

θ − SE(w)

ref ) − (SE(l)

θ − SE(l)
ref )



(cid:17)(cid:19)

We can simplify the coefficient:

Resulting in objective

b2
t
a2
t σ2
t

1 − αt
αt

1
σ2
t

=

σ2
t+1
σ2
t

≈ 1

=

(cid:18)

(cid:16)

(cid:17)(cid:19)

≈

E
DiffusionModelAlignmentUsing Direct Preference Optimization