title: Aligning Text-to-Image Diffusion Models with Reward Backpropagation
summary: Text-to-image diffusion models have recently emerged at the forefront of
  image generation, powered by very large-scale unsupervised or weakly supervised
  text-to-image training datasets. Due to their unsupervised training,
  controlling their behavior in downstream tasks, such as maximizing
  human-perceived image quality, image-text alignment, or ethical image
  generation, is difficult. Recent works finetune diffusion models to downstream
  reward functions using vanilla reinforcement learning, notorious for the high
  variance of the gradient estimators. In this paper, we propose AlignProp, a
  method that aligns diffusion models to downstream reward functions using
  end-to-end backpropagation of the reward gradient through the denoising
  process. While naive implementation of such backpropagation would require
  prohibitive memory resources for storing the partial derivatives of modern
  text-to-image models, AlignProp finetunes low-rank adapter weight modules and
  uses gradient checkpointing, to render its memory usage viable. We test
  AlignProp in finetuning diffusion models to various objectives, such as
  image-text semantic alignment, aesthetics, compressibility and controllability
  of the number of objects present, as well as their combinations. We show
  AlignProp achieves higher rewards in fewer training steps than alternatives,
  while being conceptually simpler, making it a straightforward choice for
  optimizing diffusion models for differentiable reward functions of interest.
  Code and Visualization results are available at https://align-prop.github.io/.
paper_authors._id: 651f670ee3c86280559dc39e
paper_authors.name: Mihir Prabhudesai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T08:42:55.289Z
paper_authors._id: 6310ff7dd43c55e811f8772f
paper_authors.avatarUrl: /avatars/0dcab81771329d13cc62b08e55a71c9a.svg
paper_authors.fullname: Mihir Prabhudesai
paper_authors.type: user
paper_authors.user: mihirpd
paper_authors._id: 651f670ee3c86280559dc39f
paper_authors.name: Anirudh Goyal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T08:43:03.066Z
paper_authors._id: 647d713bf812a0a790952ef9
paper_authors.avatarUrl: /avatars/1a32f43728e001a053dfdab3cab98c96.svg
paper_authors.fullname: Anirudh Goyal
paper_authors.type: user
paper_authors.user: anirudh79
paper_authors._id: 651f670ee3c86280559dc3a0
paper_authors.name: Deepak Pathak
paper_authors._id: 651f670ee3c86280559dc3a1
paper_authors.name: Katerina Fragkiadaki
paper_title: Aligning Text-to-Image Diffusion Models with Reward Backpropagation
paper_summary: Text-to-image diffusion models have recently emerged at the forefront of
  image generation, powered by very large-scale unsupervised or weakly supervised
  text-to-image training datasets. Due to their unsupervised training,
  controlling their behavior in downstream tasks, such as maximizing
  human-perceived image quality, image-text alignment, or ethical image
  generation, is difficult. Recent works finetune diffusion models to downstream
  reward functions using vanilla reinforcement learning, notorious for the high
  variance of the gradient estimators. In this paper, we propose AlignProp, a
  method that aligns diffusion models to downstream reward functions using
  end-to-end backpropagation of the reward gradient through the denoising
  process. While naive implementation of such backpropagation would require
  prohibitive memory resources for storing the partial derivatives of modern
  text-to-image models, AlignProp finetunes low-rank adapter weight modules and
  uses gradient checkpointing, to render its memory usage viable. We test
  AlignProp in finetuning diffusion models to various objectives, such as
  image-text semantic alignment, aesthetics, compressibility and controllability
  of the number of objects present, as well as their combinations. We show
  AlignProp achieves higher rewards in fewer training steps than alternatives,
  while being conceptually simpler, making it a straightforward choice for
  optimizing diffusion models for differentiable reward functions of interest.
  Code and Visualization results are available at https://align-prop.github.io/.
paper_ai_summary: AlignProp refines text-to-image diffusion models using backpropagation through the denoising process, leveraging low-rank adapters and gradient checkpointing to optimize for various objectives with higher efficiency.
paper_ai_keywords: text-to-image diffusion models reinforcement learning gradient estimators end-to-end backpropagation denoising process low-rank adapter gradient checkpointing image-text semantic alignment aesthetics compressibility number of objects conceptually simpler