title: DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion
    Models
summary: Learning from human feedback has been shown to improve text-to-image models.
  These techniques first learn a reward function that captures what humans care
  about in the task and then improve the models based on the learned reward
  function. Even though relatively simple approaches (e.g., rejection sampling
  based on reward scores) have been investigated, fine-tuning text-to-image
  models with the reward function remains challenging. In this work, we propose
  using online reinforcement learning (RL) to fine-tune text-to-image models. We
  focus on diffusion models, defining the fine-tuning task as an RL problem, and
  updating the pre-trained text-to-image diffusion models using policy gradient
  to maximize the feedback-trained reward. Our approach, coined DPOK, integrates
  policy optimization with KL regularization. We conduct an analysis of KL
  regularization for both RL fine-tuning and supervised fine-tuning. In our
  experiments, we show that DPOK is generally superior to supervised fine-tuning
  with respect to both image-text alignment and image quality.
paper_authors._id: 64741205a7f23affe77887cc
paper_authors.name: Ying Fan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T17:59:53.816Z
paper_authors._id: 64761c7595a4dc98e58c5107
paper_authors.avatarUrl: /avatars/132c907149c60c60fca77519fbd3d695.svg
paper_authors.fullname: Ying Fan
paper_authors.type: user
paper_authors.user: yingfanbot
paper_authors._id: 64741205a7f23affe77887cd
paper_authors.name: Olivia Watkins
paper_authors._id: 64741205a7f23affe77887ce
paper_authors.name: Yuqing Du
paper_authors._id: 64741205a7f23affe77887cf
paper_authors.name: Hao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-07T12:39:32.469Z
paper_authors._id: 62aa53fc5df2251f7df798c6
paper_authors.avatarUrl: /avatars/1c7828ce861c30ddd0afceff355a92ec.svg
paper_authors.fullname: Hao
paper_authors.type: user
paper_authors.user: haoliu
paper_authors._id: 64741205a7f23affe77887d0
paper_authors.name: Moonkyung Ryu
paper_authors._id: 64741205a7f23affe77887d1
paper_authors.name: Craig Boutilier
paper_authors._id: 64741205a7f23affe77887d2
paper_authors.name: Pieter Abbeel
paper_authors._id: 64741205a7f23affe77887d3
paper_authors.name: Mohammad Ghavamzadeh
paper_authors._id: 64741205a7f23affe77887d4
paper_authors.name: Kangwook Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T17:59:56.614Z
paper_authors._id: 6354137306d707b332451421
paper_authors.avatarUrl: /avatars/46770f32702e3ad08f91faeef9e4ea6e.svg
paper_authors.fullname: Kangwook Lee
paper_authors.type: user
paper_authors.user: kw1jjang
paper_authors._id: 64741205a7f23affe77887d5
paper_authors.name: Kimin Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T17:59:50.849Z
paper_authors._id: 635097ec59bfa9a85d4207b2
paper_authors.avatarUrl: /avatars/787085894e9e6538b6b3e3051efe9eea.svg
paper_authors.fullname: Kimin Lee
paper_authors.type: user
paper_authors.user: kiminle2
paper_title: DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion
    Models
paper_summary: Learning from human feedback has been shown to improve text-to-image models.
  These techniques first learn a reward function that captures what humans care
  about in the task and then improve the models based on the learned reward
  function. Even though relatively simple approaches (e.g., rejection sampling
  based on reward scores) have been investigated, fine-tuning text-to-image
  models with the reward function remains challenging. In this work, we propose
  using online reinforcement learning (RL) to fine-tune text-to-image models. We
  focus on diffusion models, defining the fine-tuning task as an RL problem, and
  updating the pre-trained text-to-image diffusion models using policy gradient
  to maximize the feedback-trained reward. Our approach, coined DPOK, integrates
  policy optimization with KL regularization. We conduct an analysis of KL
  regularization for both RL fine-tuning and supervised fine-tuning. In our
  experiments, we show that DPOK is generally superior to supervised fine-tuning
  with respect to both image-text alignment and image quality.
paper_ai_keywords: online reinforcement learning (RL) diffusion models policy gradient KL regularization image-text alignment image quality