title: Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation
summary: Fine-tuning Diffusion Models remains an underexplored frontier in generative
  artificial intelligence (GenAI), especially when compared with the remarkable
  progress made in fine-tuning Large Language Models (LLMs). While cutting-edge
  diffusion models such as Stable Diffusion (SD) and SDXL rely on supervised
  fine-tuning, their performance inevitably plateaus after seeing a certain
  volume of data. Recently, reinforcement learning (RL) has been employed to
  fine-tune diffusion models with human preference data, but it requires at least
  two images ("winner" and "loser" images) for each text prompt. In this paper,
  we introduce an innovative technique called self-play fine-tuning for diffusion
  models (SPIN-Diffusion), where the diffusion model engages in competition with
  its earlier versions, facilitating an iterative self-improvement process. Our
  approach offers an alternative to conventional supervised fine-tuning and RL
  strategies, significantly improving both model performance and alignment. Our
  experiments on the Pick-a-Pic dataset reveal that SPIN-Diffusion outperforms
  the existing supervised fine-tuning method in aspects of human preference
  alignment and visual appeal right from its first iteration. By the second
  iteration, it exceeds the performance of RLHF-based methods across all metrics,
  achieving these results with less data.
paper_authors._id: 65ced342d82f8d722c7972e6
paper_authors.name: Huizhuo Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-19T09:34:40.131Z
paper_authors._id: 657be24e8d360b690d5b665f
paper_authors.avatarUrl: /avatars/423185c3f386e3fec2e5708c1030b013.svg
paper_authors.fullname: Angela Yuan
paper_authors.type: user
paper_authors.user: angelahzyuan
paper_authors._id: 65ced342d82f8d722c7972e7
paper_authors.name: Zixiang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-16T10:36:09.971Z
paper_authors._id: 64335a225277e3b24ef3de1a
paper_authors.avatarUrl: /avatars/535fc39476b46fe6f589dcd4d7ca7f01.svg
paper_authors.fullname: Zixiang Chen
paper_authors.type: user
paper_authors.user: chenzx19
paper_authors._id: 65ced342d82f8d722c7972e8
paper_authors.name: Kaixuan Ji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-16T10:36:16.880Z
paper_authors._id: 656c1018e8bf55919a8234b0
paper_authors.avatarUrl: /avatars/0077cde47a96d91eb1d11b5a3809ead0.svg
paper_authors.fullname: Kaixuan Ji
paper_authors.type: user
paper_authors.user: KaixuanJi
paper_authors._id: 65ced342d82f8d722c7972e9
paper_authors.name: Quanquan Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-16T08:36:19.636Z
paper_authors._id: 64c039128e2612254356bba5
paper_authors.avatarUrl: /avatars/06cc76feebba0cc80ebb8f4ff86f6d9b.svg
paper_authors.fullname: Quanquan Gu
paper_authors.type: user
paper_authors.user: thughost
paper_title: Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation
paper_summary: Fine-tuning Diffusion Models remains an underexplored frontier in generative
  artificial intelligence (GenAI), especially when compared with the remarkable
  progress made in fine-tuning Large Language Models (LLMs). While cutting-edge
  diffusion models such as Stable Diffusion (SD) and SDXL rely on supervised
  fine-tuning, their performance inevitably plateaus after seeing a certain
  volume of data. Recently, reinforcement learning (RL) has been employed to
  fine-tune diffusion models with human preference data, but it requires at least
  two images ("winner" and "loser" images) for each text prompt. In this paper,
  we introduce an innovative technique called self-play fine-tuning for diffusion
  models (SPIN-Diffusion), where the diffusion model engages in competition with
  its earlier versions, facilitating an iterative self-improvement process. Our
  approach offers an alternative to conventional supervised fine-tuning and RL
  strategies, significantly improving both model performance and alignment. Our
  experiments on the Pick-a-Pic dataset reveal that SPIN-Diffusion outperforms
  the existing supervised fine-tuning method in aspects of human preference
  alignment and visual appeal right from its first iteration. By the second
  iteration, it exceeds the performance of RLHF-based methods across all metrics,
  achieving these results with less data.
paper_ai_keywords: Fine-tuning Diffusion Models Generative artificial intelligence (GenAI) Large Language Models (LLMs) Stable Diffusion (SD) SDXL Supervised fine-tuning Reinforcement learning (RL) Human preference data Self-play fine-tuning SPIN-Diffusion Iterative self-improvement Pick-a-Pic dataset Human preference alignment Visual appeal RLHF-based methods