t /σ2

t is a signal-to-noise ratio

S3. Alternate Derivation: Reinforcement Learning Perspective
We can also derive our objective as a multi-step RL approach, in the same setting as [6, 11]. A Markov Decision Process
(MDP) is a tuple (S,A, ρ0,P,R), where S is the state space, A is the action space, ρ0 is an initial state distribution, P is the
transition dynamics and R is the reward function. In this formulation, at each time step t a policy π(at|st) observes a state
st ∈ S and takes an action at ∈ A. The environment then transitions to a next state st+1 ∼ P(st+1|st, at) and the returns a
reward R(st, at).The goal of the policy is to maximize the total rewards it receives. Prior works [6, 11] map the denoising
process in diffusion model generation to this formulation via:

st ≜ (c, xt, t)

at ≜ xt

P(st+1|st, at) ≜ (δc, δt−1, δxt−1)

ρ(s0) ≜ (p(c), δT ,N (0, I))

R(st, at) =

r(c, x0)
0

if t = 0
otherwise

(cid:40)
DiffusionModelAlignmentUsing Direct Preference Optimization