Using Human Feedback to Fine-tune Diffusion Models without Any Reward
  Model
Using reinforcement learning with human feedback (RLHF) has shown significant
promise in fine-tuning diffusion models. Previous methods start by training a
reward model that aligns with human preferences, then leverage RL techniques to
fine-tune the underlying models. However, crafting an efficient reward model
demands extensive datasets, optimal architecture, and manual hyperparameter
tuning, making the process both time and cost-intensive. The direct preference
optimization (DPO) method, effective in fine-tuning large language models,
eliminates the necessity for a reward model. However, the extensive GPU memory
requirement of the diffusion model's denoising process hinders the direct
application of the DPO method. To address this issue, we introduce the Direct
Preference for Denoising Diffusion Policy Optimization (D3PO) method to
directly fine-tune diffusion models. The theoretical analysis demonstrates that
although D3PO omits training a reward model, it effectively functions as the
optimal reward model trained using human feedback data to guide the learning
process. This approach requires no training of a reward model, proving to be
more direct, cost-effective, and minimizing computational overhead. In
experiments, our method uses the relative scale of objectives as a proxy for
human preference, delivering comparable results to methods using ground-truth
rewards. Moreover, D3PO demonstrates the ability to reduce image distortion
rates and generate safer images, overcoming challenges lacking robust reward
models.
655ef17a12fb73960cf21b09
Kai Yang
claimed_verified
2023-11-24T11:41:40.779Z
64930ae54ab21070dc8a42db
/avatars/b341feaf8779b39c225c5b044f29e671.svg
Kai Yang
user
yangkaiSIGS
655ef17a12fb73960cf21b0a
Jian Tao
claimed_verified
2023-12-04T08:03:13.528Z
64671809efbd7ae3096f3e76
/avatars/1d3a9170e160fa87529619baf87fdfb5.svg
tj
user
jiantaotsing
655ef17a12fb73960cf21b0b
Jiafei Lyu
claimed_verified
2023-11-27T08:43:14.604Z
6562db314e8918182da42706
/avatars/b113bbbb496bf4dac254f0e840f08e10.svg
Jiafei Lyu
user
dmux
655ef17a12fb73960cf21b0c
Chunjiang Ge
admin_assigned
2023-11-24T09:15:04.303Z
645e054ff7a55f0d780a8ff7
/avatars/9614510443bee3bd5d6266efd1c39fc1.svg
Chunjiang Ge
user
HelloJiang
655ef17a12fb73960cf21b0d
Jiaxin Chen
655ef17a12fb73960cf21b0e
Qimai Li
655ef17a12fb73960cf21b0f
Weihan Shen
admin_assigned
2023-11-24T09:16:36.180Z
640990e9342c26884c79cc60
https://cdn-avatars.huggingface.co/v1/production/uploads/1678348475750-noauth.jpeg
WEIHAN SHEN
user
WeihanShen
655ef17a12fb73960cf21b10
Xiaolong Zhu
655ef17a12fb73960cf21b11
Xiu Li
Using Human Feedback to Fine-tune Diffusion Models without Any Reward
  Model
Using reinforcement learning with human feedback (RLHF) has shown significant
promise in fine-tuning diffusion models. Previous methods start by training a
reward model that aligns with human preferences, then leverage RL techniques to
fine-tune the underlying models. However, crafting an efficient reward model
demands extensive datasets, optimal architecture, and manual hyperparameter
tuning, making the process both time and cost-intensive. The direct preference
optimization (DPO) method, effective in fine-tuning large language models,
eliminates the necessity for a reward model. However, the extensive GPU memory
requirement of the diffusion model's denoising process hinders the direct
application of the DPO method. To address this issue, we introduce the Direct
Preference for Denoising Diffusion Policy Optimization (D3PO) method to
directly fine-tune diffusion models. The theoretical analysis demonstrates that
although D3PO omits training a reward model, it effectively functions as the
optimal reward model trained using human feedback data to guide the learning
process. This approach requires no training of a reward model, proving to be
more direct, cost-effective, and minimizing computational overhead. In
experiments, our method uses the relative scale of objectives as a proxy for
human preference, delivering comparable results to methods using ground-truth
rewards. Moreover, D3PO demonstrates the ability to reduce image distortion
rates and generate safer images, overcoming challenges lacking robust reward
models.
D3PO method directly fine-tunes diffusion models using human feedback without requiring a separate reward model, reducing computational cost and improving image quality.
reinforcement learning with human feedback (RLHF) diffusion models reward model direct preference optimization (DPO) denoising process D3PO method human preference image distortion