Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation
Fine-tuning Diffusion Models remains an underexplored frontier in generative
artificial intelligence (GenAI), especially when compared with the remarkable
progress made in fine-tuning Large Language Models (LLMs). While cutting-edge
diffusion models such as Stable Diffusion (SD) and SDXL rely on supervised
fine-tuning, their performance inevitably plateaus after seeing a certain
volume of data. Recently, reinforcement learning (RL) has been employed to
fine-tune diffusion models with human preference data, but it requires at least
two images ("winner" and "loser" images) for each text prompt. In this paper,
we introduce an innovative technique called self-play fine-tuning for diffusion
models (SPIN-Diffusion), where the diffusion model engages in competition with
its earlier versions, facilitating an iterative self-improvement process. Our
approach offers an alternative to conventional supervised fine-tuning and RL
strategies, significantly improving both model performance and alignment. Our
experiments on the Pick-a-Pic dataset reveal that SPIN-Diffusion outperforms
the existing supervised fine-tuning method in aspects of human preference
alignment and visual appeal right from its first iteration. By the second
iteration, it exceeds the performance of RLHF-based methods across all metrics,
achieving these results with less data.
65ced342d82f8d722c7972e6
Huizhuo Yuan
claimed_verified
2024-02-19T09:34:40.131Z
657be24e8d360b690d5b665f
/avatars/423185c3f386e3fec2e5708c1030b013.svg
Angela Yuan
user
angelahzyuan
65ced342d82f8d722c7972e7
Zixiang Chen
admin_assigned
2024-02-16T10:36:09.971Z
64335a225277e3b24ef3de1a
/avatars/535fc39476b46fe6f589dcd4d7ca7f01.svg
Zixiang Chen
user
chenzx19
65ced342d82f8d722c7972e8
Kaixuan Ji
admin_assigned
2024-02-16T10:36:16.880Z
656c1018e8bf55919a8234b0
/avatars/0077cde47a96d91eb1d11b5a3809ead0.svg
Kaixuan Ji
user
KaixuanJi
65ced342d82f8d722c7972e9
Quanquan Gu
claimed_verified
2024-02-16T08:36:19.636Z
64c039128e2612254356bba5
/avatars/06cc76feebba0cc80ebb8f4ff86f6d9b.svg
Quanquan Gu
user
thughost
Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation
Fine-tuning Diffusion Models remains an underexplored frontier in generative
artificial intelligence (GenAI), especially when compared with the remarkable
progress made in fine-tuning Large Language Models (LLMs). While cutting-edge
diffusion models such as Stable Diffusion (SD) and SDXL rely on supervised
fine-tuning, their performance inevitably plateaus after seeing a certain
volume of data. Recently, reinforcement learning (RL) has been employed to
fine-tune diffusion models with human preference data, but it requires at least
two images ("winner" and "loser" images) for each text prompt. In this paper,
we introduce an innovative technique called self-play fine-tuning for diffusion
models (SPIN-Diffusion), where the diffusion model engages in competition with
its earlier versions, facilitating an iterative self-improvement process. Our
approach offers an alternative to conventional supervised fine-tuning and RL
strategies, significantly improving both model performance and alignment. Our
experiments on the Pick-a-Pic dataset reveal that SPIN-Diffusion outperforms
the existing supervised fine-tuning method in aspects of human preference
alignment and visual appeal right from its first iteration. By the second
iteration, it exceeds the performance of RLHF-based methods across all metrics,
achieving these results with less data.
Self-play fine-tuning improves diffusion models' performance and alignment with human preferences using competition between model versions, outperforming supervised and RL-based methods with less data.
Diffusion Models Fine-tuning Stable Diffusion SD SDXL Supervised Fine-tuning Reinforcement Learning SPIN-Diffusion Human Preference Data Pick-a-Pic Dataset RLHF-based methods