title: Aligning Diffusion Models by Optimizing Human Utility
summary: We present Diffusion-KTO, a novel approach for aligning text-to-image
  diffusion models by formulating the alignment objective as the maximization of
  expected human utility. Since this objective applies to each generation
  independently, Diffusion-KTO does not require collecting costly pairwise
  preference data nor training a complex reward model. Instead, our objective
  requires simple per-image binary feedback signals, e.g. likes or dislikes,
  which are abundantly available. After fine-tuning using Diffusion-KTO,
  text-to-image diffusion models exhibit superior performance compared to
  existing techniques, including supervised fine-tuning and Diffusion-DPO, both
  in terms of human judgment and automatic evaluation metrics such as PickScore
  and ImageReward. Overall, Diffusion-KTO unlocks the potential of leveraging
  readily available per-image binary signals and broadens the applicability of
  aligning text-to-image diffusion models with human preferences.
paper_authors._id: 6614c4b0c13e5d702f5b6ca2
paper_authors.name: Shufan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:47:40.616Z
paper_authors._id: 6310531914aa81e1044363ed
paper_authors.avatarUrl: /avatars/ae7767e591cb7199ea2f62d2db89fc7f.svg
paper_authors.fullname: Shufan Li
paper_authors.type: user
paper_authors.user: jacklishufan
paper_authors._id: 6614c4b0c13e5d702f5b6ca3
paper_authors.name: Konstantinos Kallidromitis
paper_authors._id: 6614c4b0c13e5d702f5b6ca4
paper_authors.name: Akash Gokul
paper_authors._id: 6614c4b0c13e5d702f5b6ca5
paper_authors.name: Yusuke Kato
paper_authors._id: 6614c4b0c13e5d702f5b6ca6
paper_authors.name: Kazuki Kozuka
paper_title: Aligning Diffusion Models by Optimizing Human Utility
paper_summary: We present Diffusion-KTO, a novel approach for aligning text-to-image
  diffusion models by formulating the alignment objective as the maximization of
  expected human utility. Since this objective applies to each generation
  independently, Diffusion-KTO does not require collecting costly pairwise
  preference data nor training a complex reward model. Instead, our objective
  requires simple per-image binary feedback signals, e.g. likes or dislikes,
  which are abundantly available. After fine-tuning using Diffusion-KTO,
  text-to-image diffusion models exhibit superior performance compared to
  existing techniques, including supervised fine-tuning and Diffusion-DPO, both
  in terms of human judgment and automatic evaluation metrics such as PickScore
  and ImageReward. Overall, Diffusion-KTO unlocks the potential of leveraging
  readily available per-image binary signals and broadens the applicability of
  aligning text-to-image diffusion models with human preferences.
paper_ai_summary: Diffusion-KTO fine-tunes text-to-image diffusion models using per-image binary feedback, enhancing performance in both human and automatic evaluations.
paper_ai_keywords: Diffusion-KTO text-to-image diffusion models alignment objective human utility pairwise preference data reward model per-image binary feedback signals supervised fine-tuning Diffusion-DPO PickScore ImageReward