title: Diffusion Model Alignment Using Direct Preference Optimization
summary: Large language models (LLMs) are fine-tuned using human comparison data with
  Reinforcement Learning from Human Feedback (RLHF) methods to make them better
  aligned with users' preferences. In contrast to LLMs, human preference learning
  has not been widely explored in text-to-image diffusion models; the best
  existing approach is to fine-tune a pretrained model using carefully curated
  high quality images and captions to improve visual appeal and text alignment.
  We propose Diffusion-DPO, a method to align diffusion models to human
  preferences by directly optimizing on human comparison data. Diffusion-DPO is
  adapted from the recently developed Direct Preference Optimization (DPO), a
  simpler alternative to RLHF which directly optimizes a policy that best
  satisfies human preferences under a classification objective. We re-formulate
  DPO to account for a diffusion model notion of likelihood, utilizing the
  evidence lower bound to derive a differentiable objective. Using the Pick-a-Pic
  dataset of 851K crowdsourced pairwise preferences, we fine-tune the base model
  of the state-of-the-art Stable Diffusion XL (SDXL)-1.0 model with
  Diffusion-DPO. Our fine-tuned base model significantly outperforms both base
  SDXL-1.0 and the larger SDXL-1.0 model consisting of an additional refinement
  model in human evaluation, improving visual appeal and prompt alignment. We
  also develop a variant that uses AI feedback and has comparable performance to
  training on human preferences, opening the door for scaling of diffusion model
  alignment methods.
paper_authors._id: 655ee5917b44c113b8367213
paper_authors.name: Bram Wallace
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:39:25.309Z
paper_authors._id: 62965a7b7d586f66c392221b
paper_authors.avatarUrl: /avatars/c5f17cc83646aa2c2101c91f4eca138d.svg
paper_authors.fullname: Bram Wallace
paper_authors.type: user
paper_authors.user: bramw
paper_authors._id: 655ee5917b44c113b8367214
paper_authors.name: Meihua Dang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:39:34.028Z
paper_authors._id: 652eccd3aeb9826ab9cef207
paper_authors.avatarUrl: /avatars/2172ec9a2e5f94f006c9a4b8afa01112.svg
paper_authors.fullname: Meihua Dang
paper_authors.type: user
paper_authors.user: mhdang
paper_authors._id: 655ee5917b44c113b8367215
paper_authors.name: Rafael Rafailov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:39:41.735Z
paper_authors._id: 64701c98d742e9ef6522598f
paper_authors.avatarUrl: /avatars/df3a52bd4bc374d7cb4061f449f75c07.svg
paper_authors.fullname: Rafael Rafailov
paper_authors.type: user
paper_authors.user: rmrafailov
paper_authors._id: 655ee5917b44c113b8367216
paper_authors.name: Linqi Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:39:49.966Z
paper_authors._id: 64554c3c9d37c3fb332975ed
paper_authors.avatarUrl: /avatars/13001d48b56cca99471a7e75116b70cf.svg
paper_authors.fullname: Linqi Zhou
paper_authors.type: user
paper_authors.user: alexzhou907
paper_authors._id: 655ee5917b44c113b8367217
paper_authors.name: Aaron Lou
paper_authors._id: 655ee5917b44c113b8367218
paper_authors.name: Senthil Purushwalkam
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:40:15.660Z
paper_authors._id: 637e633366a51ca5d960a9b1
paper_authors.avatarUrl: /avatars/e53242e4e36875105be087c46a552e29.svg
paper_authors.fullname: Senthil Purushwalkam
paper_authors.type: user
paper_authors.user: senthilps8
paper_authors._id: 655ee5917b44c113b8367219
paper_authors.name: Stefano Ermon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:40:22.750Z
paper_authors._id: 62f6e244329d4d014d1f4ac5
paper_authors.avatarUrl: /avatars/5a8b2bb063c2ebc340504b22530f6811.svg
paper_authors.fullname: Stefano Ermon
paper_authors.type: user
paper_authors.user: ermonste
paper_authors._id: 655ee5917b44c113b836721a
paper_authors.name: Caiming Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:40:29.708Z
paper_authors._id: 649dbcc4e0fff1ed099dc80a
paper_authors.avatarUrl: /avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
paper_authors.fullname: Caiming Xiong
paper_authors.type: user
paper_authors.user: cxiong
paper_authors._id: 655ee5917b44c113b836721b
paper_authors.name: Shafiq Joty
paper_authors._id: 655ee5917b44c113b836721c
paper_authors.name: Nikhil Naik
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:40:49.932Z
paper_authors._id: 627c2c751b02d7b5bf663fb0
paper_authors.avatarUrl: /avatars/f33ac735e2cc910d2da1286d476de5db.svg
paper_authors.fullname: Nikhil Naik
paper_authors.type: user
paper_authors.user: nikdnaik
paper_title: Diffusion Model Alignment Using Direct Preference Optimization
paper_summary: Large language models (LLMs) are fine-tuned using human comparison data with
  Reinforcement Learning from Human Feedback (RLHF) methods to make them better
  aligned with users' preferences. In contrast to LLMs, human preference learning
  has not been widely explored in text-to-image diffusion models; the best
  existing approach is to fine-tune a pretrained model using carefully curated
  high quality images and captions to improve visual appeal and text alignment.
  We propose Diffusion-DPO, a method to align diffusion models to human
  preferences by directly optimizing on human comparison data. Diffusion-DPO is
  adapted from the recently developed Direct Preference Optimization (DPO), a
  simpler alternative to RLHF which directly optimizes a policy that best
  satisfies human preferences under a classification objective. We re-formulate
  DPO to account for a diffusion model notion of likelihood, utilizing the
  evidence lower bound to derive a differentiable objective. Using the Pick-a-Pic
  dataset of 851K crowdsourced pairwise preferences, we fine-tune the base model
  of the state-of-the-art Stable Diffusion XL (SDXL)-1.0 model with
  Diffusion-DPO. Our fine-tuned base model significantly outperforms both base
  SDXL-1.0 and the larger SDXL-1.0 model consisting of an additional refinement
  model in human evaluation, improving visual appeal and prompt alignment. We
  also develop a variant that uses AI feedback and has comparable performance to
  training on human preferences, opening the door for scaling of diffusion model
  alignment methods.
paper_ai_keywords: Reinforcement Learning from Human Feedback (RLHF) text-to-image diffusion models Direct Preference Optimization (DPO) evidence lower bound differentiable objective Pick-a-Pic dataset Stable Diffusion XL (SDXL)-1.0 prompt alignment AI feedback