title: Using Human Feedback to Fine-tune Diffusion Models without Any Reward
    Model
summary: Using reinforcement learning with human feedback (RLHF) has shown significant
  promise in fine-tuning diffusion models. Previous methods start by training a
  reward model that aligns with human preferences, then leverage RL techniques to
  fine-tune the underlying models. However, crafting an efficient reward model
  demands extensive datasets, optimal architecture, and manual hyperparameter
  tuning, making the process both time and cost-intensive. The direct preference
  optimization (DPO) method, effective in fine-tuning large language models,
  eliminates the necessity for a reward model. However, the extensive GPU memory
  requirement of the diffusion model's denoising process hinders the direct
  application of the DPO method. To address this issue, we introduce the Direct
  Preference for Denoising Diffusion Policy Optimization (D3PO) method to
  directly fine-tune diffusion models. The theoretical analysis demonstrates that
  although D3PO omits training a reward model, it effectively functions as the
  optimal reward model trained using human feedback data to guide the learning
  process. This approach requires no training of a reward model, proving to be
  more direct, cost-effective, and minimizing computational overhead. In
  experiments, our method uses the relative scale of objectives as a proxy for
  human preference, delivering comparable results to methods using ground-truth
  rewards. Moreover, D3PO demonstrates the ability to reduce image distortion
  rates and generate safer images, overcoming challenges lacking robust reward
  models.
paper_authors._id: 655ef17a12fb73960cf21b09
paper_authors.name: Kai Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-24T11:41:40.779Z
paper_authors._id: 64930ae54ab21070dc8a42db
paper_authors.avatarUrl: /avatars/b341feaf8779b39c225c5b044f29e671.svg
paper_authors.fullname: Kai Yang
paper_authors.type: user
paper_authors.user: yangkaiSIGS
paper_authors._id: 655ef17a12fb73960cf21b0a
paper_authors.name: Jian Tao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-04T08:03:13.528Z
paper_authors._id: 64671809efbd7ae3096f3e76
paper_authors.avatarUrl: /avatars/1d3a9170e160fa87529619baf87fdfb5.svg
paper_authors.fullname: tj
paper_authors.type: user
paper_authors.user: jiantaotsing
paper_authors._id: 655ef17a12fb73960cf21b0b
paper_authors.name: Jiafei Lyu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-27T08:43:14.604Z
paper_authors._id: 6562db314e8918182da42706
paper_authors.avatarUrl: /avatars/b113bbbb496bf4dac254f0e840f08e10.svg
paper_authors.fullname: Jiafei Lyu
paper_authors.type: user
paper_authors.user: dmux
paper_authors._id: 655ef17a12fb73960cf21b0c
paper_authors.name: Chunjiang Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-24T09:15:04.303Z
paper_authors._id: 645e054ff7a55f0d780a8ff7
paper_authors.avatarUrl: /avatars/9614510443bee3bd5d6266efd1c39fc1.svg
paper_authors.fullname: Chunjiang Ge
paper_authors.type: user
paper_authors.user: HelloJiang
paper_authors._id: 655ef17a12fb73960cf21b0d
paper_authors.name: Jiaxin Chen
paper_authors._id: 655ef17a12fb73960cf21b0e
paper_authors.name: Qimai Li
paper_authors._id: 655ef17a12fb73960cf21b0f
paper_authors.name: Weihan Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-24T09:16:36.180Z
paper_authors._id: 640990e9342c26884c79cc60
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678348475750-noauth.jpeg
paper_authors.fullname: WEIHAN SHEN
paper_authors.type: user
paper_authors.user: WeihanShen
paper_authors._id: 655ef17a12fb73960cf21b10
paper_authors.name: Xiaolong Zhu
paper_authors._id: 655ef17a12fb73960cf21b11
paper_authors.name: Xiu Li
paper_title: Using Human Feedback to Fine-tune Diffusion Models without Any Reward
    Model
paper_summary: Using reinforcement learning with human feedback (RLHF) has shown significant
  promise in fine-tuning diffusion models. Previous methods start by training a
  reward model that aligns with human preferences, then leverage RL techniques to
  fine-tune the underlying models. However, crafting an efficient reward model
  demands extensive datasets, optimal architecture, and manual hyperparameter
  tuning, making the process both time and cost-intensive. The direct preference
  optimization (DPO) method, effective in fine-tuning large language models,
  eliminates the necessity for a reward model. However, the extensive GPU memory
  requirement of the diffusion model's denoising process hinders the direct
  application of the DPO method. To address this issue, we introduce the Direct
  Preference for Denoising Diffusion Policy Optimization (D3PO) method to
  directly fine-tune diffusion models. The theoretical analysis demonstrates that
  although D3PO omits training a reward model, it effectively functions as the
  optimal reward model trained using human feedback data to guide the learning
  process. This approach requires no training of a reward model, proving to be
  more direct, cost-effective, and minimizing computational overhead. In
  experiments, our method uses the relative scale of objectives as a proxy for
  human preference, delivering comparable results to methods using ground-truth
  rewards. Moreover, D3PO demonstrates the ability to reduce image distortion
  rates and generate safer images, overcoming challenges lacking robust reward
  models.
paper_ai_summary: D3PO method directly fine-tunes diffusion models using human feedback without requiring a separate reward model, reducing computational cost and improving image quality.
paper_ai_keywords: reinforcement learning with human feedback (RLHF) diffusion models reward model direct preference optimization (DPO) denoising process D3PO method human preference image distortion