1. Introduction

Text-to-image diffusion models have been the state-of-the-
art in image generation for the past few years. They are typ-
ically trained in a single stage, using web-scale datasets of
text-image pairs by applying the diffusion objective. This

stands in contrast to the state-of-the-art training method-
ology for Large Language Models (LLMs). The best per-
forming LLMs [28, 48] are trained in two stages.
In the
first (“pretraining”) stage, they are trained on large web-
scale data.
In the second (“alignment”) stage, they are
fine-tuned to make them better aligned with human pref-
erences. Alignment is typically performed using super-
vised fine-tuning (SFT) and Reinforcement Learning from
Human Feedback (RLHF) using preference data. LLMs
trained with this two-stage process have set the state-of-the-
art in language generation tasks and have been deployed in
commercial applications such as ChatGPT and Bard.
DiffusionModelAlignmentUsing Direct Preference Optimization