0 , xB

0 ) with the same conditioning c we can estimate the reward difference using Eq. (48).

When doing this it improves the estimate to average over multiple draws (t, qt). We use this method in Table S2.

S5. β Ablation

Figure S2. Median PickScores for generations on the Pick-a-Pic v2 validation set for different choices of β

For β far below the displayed regime, the diffusion model degenerates into a pure reward scoring model. Much greater,

and the KL-divergence penalty greatly restricts any appreciable adaptation.
DiffusionModelAlignmentUsing Direct Preference Optimization