title: Scalable Ranked Preference Optimization for Text-to-Image Generation
summary: Direct Preference Optimization (DPO) has emerged as a powerful approach to
  align text-to-image (T2I) models with human feedback. Unfortunately, successful
  application of DPO to T2I models requires a huge amount of resources to collect
  and label large-scale datasets, e.g., millions of generated paired images
  annotated with human preferences. In addition, these human preference datasets
  can get outdated quickly as the rapid improvements of T2I models lead to higher
  quality images. In this work, we investigate a scalable approach for collecting
  large-scale and fully synthetic datasets for DPO training. Specifically, the
  preferences for paired images are generated using a pre-trained reward
  function, eliminating the need for involving humans in the annotation process,
  greatly improving the dataset collection efficiency. Moreover, we demonstrate
  that such datasets allow averaging predictions across multiple models and
  collecting ranked preferences as opposed to pairwise preferences. Furthermore,
  we introduce RankDPO to enhance DPO-based methods using the ranking feedback.
  Applying RankDPO on SDXL and SD3-Medium models with our synthetically generated
  preference dataset ``Syn-Pic'' improves both prompt-following (on benchmarks
  like T2I-Compbench, GenEval, and DPG-Bench) and visual quality (through user
  studies). This pipeline presents a practical and scalable solution to develop
  better preference datasets to enhance the performance of text-to-image models.
paper_authors._id: 671a01b4c673da43e82b2867
paper_authors.name: Shyamgopal Karthik
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-24T08:31:42.279Z
paper_authors._id: 6254599b6e36fe62e141c8f9
paper_authors.avatarUrl: /avatars/08d6b68b92c7bfd0a8be022ba9f2f289.svg
paper_authors.fullname: Shyamgopal Karthik
paper_authors.type: user
paper_authors.user: shyamgopal
paper_authors._id: 671a01b4c673da43e82b2868
paper_authors.name: Huseyin Coskun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T08:35:13.318Z
paper_authors._id: 64eaefcd5349043b2b9670a2
paper_authors.avatarUrl: /avatars/9ca92fdda3f7b14586f40e9d31a746fe.svg
paper_authors.fullname: Huseyin Coskun
paper_authors.type: user
paper_authors.user: hcoskun
paper_authors._id: 671a01b4c673da43e82b2869
paper_authors.name: Zeynep Akata
paper_authors._id: 671a01b4c673da43e82b286a
paper_authors.name: Sergey Tulyakov
paper_authors._id: 671a01b4c673da43e82b286b
paper_authors.name: Jian Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-15T08:49:46.801Z
paper_authors._id: 61f19829233c91cbd2f79e70
paper_authors.avatarUrl: /avatars/a0735a94542b4f7cda5aed8bc4be0538.svg
paper_authors.fullname: Jian Ren
paper_authors.type: user
paper_authors.user: alanspike
paper_authors._id: 671a01b4c673da43e82b286c
paper_authors.name: Anil Kag
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T08:35:38.040Z
paper_authors._id: 66b01ee8e53bbad918362856
paper_authors.avatarUrl: /avatars/293529589a91dd7a95909d66727db224.svg
paper_authors.fullname: Anil Kag
paper_authors.type: user
paper_authors.user: anilkagak2
paper_title: Scalable Ranked Preference Optimization for Text-to-Image Generation
paper_summary: Direct Preference Optimization (DPO) has emerged as a powerful approach to
  align text-to-image (T2I) models with human feedback. Unfortunately, successful
  application of DPO to T2I models requires a huge amount of resources to collect
  and label large-scale datasets, e.g., millions of generated paired images
  annotated with human preferences. In addition, these human preference datasets
  can get outdated quickly as the rapid improvements of T2I models lead to higher
  quality images. In this work, we investigate a scalable approach for collecting
  large-scale and fully synthetic datasets for DPO training. Specifically, the
  preferences for paired images are generated using a pre-trained reward
  function, eliminating the need for involving humans in the annotation process,
  greatly improving the dataset collection efficiency. Moreover, we demonstrate
  that such datasets allow averaging predictions across multiple models and
  collecting ranked preferences as opposed to pairwise preferences. Furthermore,
  we introduce RankDPO to enhance DPO-based methods using the ranking feedback.
  Applying RankDPO on SDXL and SD3-Medium models with our synthetically generated
  preference dataset ``Syn-Pic'' improves both prompt-following (on benchmarks
  like T2I-Compbench, GenEval, and DPG-Bench) and visual quality (through user
  studies). This pipeline presents a practical and scalable solution to develop
  better preference datasets to enhance the performance of text-to-image models.
paper_ai_keywords: Direct Preference Optimization (DPO) text-to-image (T2I) human feedback T2I models large-scale datasets human preferences generated paired images pre-trained reward function dataset collection efficiency ranking feedback RankDPO prompt-following benchmarks (T2I-Compbench, GenEval, DPG-Bench) visual quality user studies preference datasets Syn-Pic SDXL SD3-Medium