Aligning Diffusion Models Alignment of diffusion mod-
els to human preferences has so far been much less explored
than in the case of LLMs. Multiple approaches [30, 36]
fine-tune on datasets scored as highly visually appealing by
an aesthetics classifier [37], to bias the model to visually ap-
pealing generations. Emu [9] finetunes a pretrained model
using a small, curated image dataset of high quality pho-
tographs with manually written detailed captions to improve
visual appeal and text alignment. Other methods [15, 39] re-
caption existing web-scraped image datasets to improve text
fidelity. Caption-aware human preference scoring models
are trained on generation preference datasets [21, 52, 55],
but the impact of these reward models to the generative
space has been limited. DOODL [51] introduces the task
of aesthetically improving a single generation iteratively at
inference time. DRAFT [7] and AlignProp [31], incorpo-
rate a similar approach into training: tuning the generative
DiffusionModelAlignmentUsing Direct Preference Optimization