Ethics The performance of Diffusion-DPO is impres-
sive, but any effort in text-to-image generation presents
ethical risks, particularly when data are web-collected.
Generations of harmful, hateful, fake or sexually explicit
content are known risk vectors. Beyond that, this approach
is increasingly subject to the biases of the participating
labelers (in addition to the biases present in the pretrained
model); Diffusion-DPO can learn and propagate these
preferences. As a result, a diverse and representative
set of labelers is essential – whose preferences in turn
become encoded in the dataset. Furthermore, a portion
of user-generated Pick-a-Pic prompts are overtly sexual,
and even innocuous prompts may deliver images that
skew more suggestively (particularly for prompts that
hyper-sexualize women). Finally, as with all text-to-image
models, the image produced will not always match the
prompt. Hearteningly though, some of these scenarios
DiffusionModelAlignmentUsing Direct Preference Optimization