Large language models (LLMs) are fine-tuned using hu-
man comparison data with Reinforcement Learning from
Human Feedback (RLHF) methods to make them better
aligned with users’ preferences.
In contrast to LLMs,
human preference learning has not been widely explored
in text-to-image diffusion models; the best existing ap-
proach is to fine-tune a pretrained model using carefully
curated high quality images and captions to improve vi-
sual appeal and text alignment. We propose Diffusion-
DPO, a method to align diffusion models to human pref-
erences by directly optimizing on human comparison data.
Diffusion-DPO is adapted from the recently developed Di-
rect Preference Optimization (DPO) [33], a simpler al-
ternative to RLHF which directly optimizes a policy that
best satisfies human preferences under a classification ob-
jective. We re-formulate DPO to account for a diffusion
model notion of likelihood, utilizing the evidence lower
DiffusionModelAlignmentUsing Direct Preference Optimization