Aligning Diffusion Models by Optimizing Human Utility
<|reference_start|>Aligning Diffusion Models by Optimizing Human Utility: We present Diffusion-KTO, a novel approach for aligning text-to-image diffusion models by formulating the alignment objective as the maximization of expected human utility. Since this objective applies to each generation independently, Diffusion-KTO does not require collecting costly pairwise preference data nor training a complex reward model. Instead, our objective requires simple per-image binary feedback signals, e.g. likes or dislikes, which are abundantly available. After fine-tuning using Diffusion-KTO, text-to-image diffusion models exhibit superior performance compared to existing techniques, including supervised fine-tuning and Diffusion-DPO, both in terms of human judgment and automatic evaluation metrics such as PickScore and ImageReward. Overall, Diffusion-KTO unlocks the potential of leveraging readily available per-image binary signals and broadens the applicability of aligning text-to-image diffusion models with human preferences.<|reference_end|>
@article{li2024aligning,
    title={Aligning Diffusion Models by Optimizing Human Utility},
    author={Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato,
  Kazuki Kozuka},
    journal={arXiv preprint arXiv:2404.04465},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.04465},
    primaryClass={cs.CV}
}