title: Training Diffusion Models with Reinforcement Learning
summary: Diffusion models are a class of flexible generative models trained with an
  approximation to the log-likelihood objective. However, most use cases of
  diffusion models are not concerned with likelihoods, but instead with
  downstream objectives such as human-perceived image quality or drug
  effectiveness. In this paper, we investigate reinforcement learning methods for
  directly optimizing diffusion models for such objectives. We describe how
  posing denoising as a multi-step decision-making problem enables a class of
  policy gradient algorithms, which we refer to as denoising diffusion policy
  optimization (DDPO), that are more effective than alternative reward-weighted
  likelihood approaches. Empirically, DDPO is able to adapt text-to-image
  diffusion models to objectives that are difficult to express via prompting,
  such as image compressibility, and those derived from human feedback, such as
  aesthetic quality. Finally, we show that DDPO can improve prompt-image
  alignment using feedback from a vision-language model without the need for
  additional data collection or human annotation.
paper_authors._id: 646c1548e5abcbf670a39f0e
paper_authors.name: Kevin Black
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T20:14:43.276Z
paper_authors._id: 646fde2c799a974be31b6031
paper_authors.avatarUrl: /avatars/d816aa2d18930f4a634096f1c82a49bf.svg
paper_authors.fullname: Kevin Black
paper_authors.type: user
paper_authors.user: kvablack
paper_authors._id: 646c1548e5abcbf670a39f0f
paper_authors.name: Michael Janner
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T08:32:10.698Z
paper_authors._id: 6317f8c7aea37e1333e3d476
paper_authors.avatarUrl: /avatars/a2eb33dafc3b475e02a1ad182a964c1b.svg
paper_authors.fullname: Michael Janner
paper_authors.type: user
paper_authors.user: janner
paper_authors._id: 646c1548e5abcbf670a39f10
paper_authors.name: Yilun Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T08:31:34.506Z
paper_authors._id: 63c9bd445fdc575773c732fe
paper_authors.avatarUrl: /avatars/def472d1ab3fbf751225357c0932ae7e.svg
paper_authors.fullname: Yilun Du
paper_authors.type: user
paper_authors.user: yilundu
paper_authors._id: 646c1548e5abcbf670a39f11
paper_authors.name: Ilya Kostrikov
paper_authors._id: 646c1548e5abcbf670a39f12
paper_authors.name: Sergey Levine
paper_title: Training Diffusion Models with Reinforcement Learning
paper_summary: Diffusion models are a class of flexible generative models trained with an
  approximation to the log-likelihood objective. However, most use cases of
  diffusion models are not concerned with likelihoods, but instead with
  downstream objectives such as human-perceived image quality or drug
  effectiveness. In this paper, we investigate reinforcement learning methods for
  directly optimizing diffusion models for such objectives. We describe how
  posing denoising as a multi-step decision-making problem enables a class of
  policy gradient algorithms, which we refer to as denoising diffusion policy
  optimization (DDPO), that are more effective than alternative reward-weighted
  likelihood approaches. Empirically, DDPO is able to adapt text-to-image
  diffusion models to objectives that are difficult to express via prompting,
  such as image compressibility, and those derived from human feedback, such as
  aesthetic quality. Finally, we show that DDPO can improve prompt-image
  alignment using feedback from a vision-language model without the need for
  additional data collection or human annotation.
paper_ai_keywords: diffusion models log-likelihood objective human-perceived image quality drug effectiveness reinforcement learning policy gradient algorithms denoising diffusion policy optimization (DDPO) prompt-image alignment vision-language model