L1(θ) ≤ −Et,ϵw,ϵl log σ (−βT ω(λt) ( ∥ϵw−ϵθ(xw

t , t)∥2−∥ϵw−ϵref(xw

t , t)∥2 −(cid:0)∥ϵl − ϵθ(xl

where ϵw, ϵl ∼ N (0, I), xt ∼ q(xt|x0) thus xt = αtx0 + σtϵ. Same as Eq. (2), λt = α2
term [20], in practice, the reweighting assigns each term the same weight [17].
An alternative approximation Note that for Eq. (18) we utilize q(x1:T|x0) to approximate pθ(x1:T|x0). For each
step, it is to use q(xt−1,t|x0) to approximate pθ(xt−1,t|x0). Alternatively, we also propose to use q(xt|x0)pθ(xt−1|xt)
for approximation. And this approximation yields lower error because DKL(q(xt|x0)pθ(xt−1|xt)∥pθ(xt−1,t|x0)) =
DKL(q(xt|x0)∥pθ(xt|x0)) < DKL(q(xt−1,t|x0)∥pθ(xt−1,t|x0)).

t /σ2

t, t)∥2 − ∥ϵl − ϵref(xl
t is a signal-to-noise ratio

LDPO-Diffusion(θ) = − log σ

βE

1:T ∼pθ(x1:T |xw
xw

0 ),xl

1:T ∼pθ(x1:T |xl
0)

= − log σ

βE

1:T ∼pθ(x1:T |xw
xw

0 ),xl

1:T ∼pθ(x1:T |xl
0)

log

= − log σ

βE

1:T ∼pθ(x1:T |xw
xw

0 ),xl

1:T ∼pθ(x1:T |xl

0)T Et

log

= − log σ

βT EtE

t−1,t∼pθ(xt−1,t|xw
xw
DiffusionModelAlignmentUsing Direct Preference Optimization