title: Large-scale Reinforcement Learning for Diffusion Models
summary: Text-to-image diffusion models are a class of deep generative models that
  have demonstrated an impressive capacity for high-quality image generation.
  However, these models are susceptible to implicit biases that arise from
  web-scale text-image training pairs and may inaccurately model aspects of
  images we care about. This can result in suboptimal samples, model bias, and
  images that do not align with human ethics and preferences. In this paper, we
  present an effective scalable algorithm to improve diffusion models using
  Reinforcement Learning (RL) across a diverse set of reward functions, such as
  human preference, compositionality, and fairness over millions of images. We
  illustrate how our approach substantially outperforms existing methods for
  aligning diffusion models with human preferences. We further illustrate how
  this substantially improves pretrained Stable Diffusion (SD) models, generating
  samples that are preferred by humans 80.3% of the time over those from the base
  SD model while simultaneously improving both the composition and diversity of
  generated samples.
paper_authors._id: 65b071e07febbcc2afc9d680
paper_authors.name: Yinan Zhang
paper_authors._id: 65b071e07febbcc2afc9d681
paper_authors.name: Eric Tzeng
paper_authors._id: 65b071e07febbcc2afc9d682
paper_authors.name: Yilun Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T09:15:33.839Z
paper_authors._id: 63c9bd445fdc575773c732fe
paper_authors.avatarUrl: /avatars/def472d1ab3fbf751225357c0932ae7e.svg
paper_authors.fullname: Yilun Du
paper_authors.type: user
paper_authors.user: yilundu
paper_authors._id: 65b071e07febbcc2afc9d683
paper_authors.name: Dmitry Kislyuk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T09:15:40.490Z
paper_authors._id: 6383ce4411cd91c15d5c9a11
paper_authors.avatarUrl: /avatars/aba18939f4744185f0b5fb9a1ba2ce03.svg
paper_authors.fullname: Dmitry Kislyuk
paper_authors.type: user
paper_authors.user: molkie
paper_title: Large-scale Reinforcement Learning for Diffusion Models
paper_summary: Text-to-image diffusion models are a class of deep generative models that
  have demonstrated an impressive capacity for high-quality image generation.
  However, these models are susceptible to implicit biases that arise from
  web-scale text-image training pairs and may inaccurately model aspects of
  images we care about. This can result in suboptimal samples, model bias, and
  images that do not align with human ethics and preferences. In this paper, we
  present an effective scalable algorithm to improve diffusion models using
  Reinforcement Learning (RL) across a diverse set of reward functions, such as
  human preference, compositionality, and fairness over millions of images. We
  illustrate how our approach substantially outperforms existing methods for
  aligning diffusion models with human preferences. We further illustrate how
  this substantially improves pretrained Stable Diffusion (SD) models, generating
  samples that are preferred by humans 80.3% of the time over those from the base
  SD model while simultaneously improving both the composition and diversity of
  generated samples.
paper_ai_keywords: text-to-image diffusion models deep generative models web-scale text-image training pairs implicit biases Reinforcement Learning (RL) reward functions human preference compositionality fairness pretrained Stable Diffusion (SD) models