title: PRDP: Proximal Reward Difference Prediction for Large-Scale Reward
    Finetuning of Diffusion Models
summary: Reward finetuning has emerged as a promising approach to aligning foundation
  models with downstream objectives. Remarkable success has been achieved in the
  language domain by using reinforcement learning (RL) to maximize rewards that
  reflect human preference. However, in the vision domain, existing RL-based
  reward finetuning methods are limited by their instability in large-scale
  training, rendering them incapable of generalizing to complex, unseen prompts.
  In this paper, we propose Proximal Reward Difference Prediction (PRDP),
  enabling stable black-box reward finetuning for diffusion models for the first
  time on large-scale prompt datasets with over 100K prompts. Our key innovation
  is the Reward Difference Prediction (RDP) objective that has the same optimal
  solution as the RL objective while enjoying better training stability.
  Specifically, the RDP objective is a supervised regression objective that tasks
  the diffusion model with predicting the reward difference of generated image
  pairs from their denoising trajectories. We theoretically prove that the
  diffusion model that obtains perfect reward difference prediction is exactly
  the maximizer of the RL objective. We further develop an online algorithm with
  proximal updates to stably optimize the RDP objective. In experiments, we
  demonstrate that PRDP can match the reward maximization ability of
  well-established RL-based methods in small-scale training. Furthermore, through
  large-scale training on text prompts from the Human Preference Dataset v2 and
  the Pick-a-Pic v1 dataset, PRDP achieves superior generation quality on a
  diverse set of complex, unseen prompts whereas RL-based methods completely
  fail.
paper_authors._id: 65cda11815927632e844a903
paper_authors.name: Fei Deng
paper_authors._id: 65cda11815927632e844a904
paper_authors.name: Qifei Wang
paper_authors._id: 65cda11815927632e844a905
paper_authors.name: Wei Wei
paper_authors._id: 65cda11815927632e844a906
paper_authors.name: Matthias Grundmann
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-15T10:03:53.191Z
paper_authors._id: 60fa5a5f8b6cceb9a84297d0
paper_authors.avatarUrl: /avatars/3a59609db54687cc2e1e373b897e7f5c.svg
paper_authors.fullname: Matthias Grundmann
paper_authors.type: user
paper_authors.user: grundman
paper_authors._id: 65cda11815927632e844a907
paper_authors.name: Tingbo Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-15T10:04:02.476Z
paper_authors._id: 655846d7ed8df83128f5826a
paper_authors.avatarUrl: /avatars/d7ce174d7d1b8614d5f6f071225c0057.svg
paper_authors.fullname: Hou
paper_authors.type: user
paper_authors.user: Tingbo
paper_title: PRDP: Proximal Reward Difference Prediction for Large-Scale Reward
    Finetuning of Diffusion Models
paper_summary: Reward finetuning has emerged as a promising approach to aligning foundation
  models with downstream objectives. Remarkable success has been achieved in the
  language domain by using reinforcement learning (RL) to maximize rewards that
  reflect human preference. However, in the vision domain, existing RL-based
  reward finetuning methods are limited by their instability in large-scale
  training, rendering them incapable of generalizing to complex, unseen prompts.
  In this paper, we propose Proximal Reward Difference Prediction (PRDP),
  enabling stable black-box reward finetuning for diffusion models for the first
  time on large-scale prompt datasets with over 100K prompts. Our key innovation
  is the Reward Difference Prediction (RDP) objective that has the same optimal
  solution as the RL objective while enjoying better training stability.
  Specifically, the RDP objective is a supervised regression objective that tasks
  the diffusion model with predicting the reward difference of generated image
  pairs from their denoising trajectories. We theoretically prove that the
  diffusion model that obtains perfect reward difference prediction is exactly
  the maximizer of the RL objective. We further develop an online algorithm with
  proximal updates to stably optimize the RDP objective. In experiments, we
  demonstrate that PRDP can match the reward maximization ability of
  well-established RL-based methods in small-scale training. Furthermore, through
  large-scale training on text prompts from the Human Preference Dataset v2 and
  the Pick-a-Pic v1 dataset, PRDP achieves superior generation quality on a
  diverse set of complex, unseen prompts whereas RL-based methods completely
  fail.
paper_ai_summary: Proximal Reward Difference Prediction enables stable large-scale reward finetuning of diffusion models in vision, outperforming RL-based methods on complex prompts.
paper_ai_keywords: reward finetuning reinforcement learning diffusion models reward difference prediction denoising trajectories Proximal Reward Difference Prediction Human Preference Dataset Pick-a-Pic Dataset