title: New Desiderata for Direct Preference Optimization
summary: Large language models in the past have typically relied on some form of
  reinforcement learning with human feedback (RLHF) to better align model
  responses with human preferences. However, because of oft-observed
  instabilities when implementing these RLHF pipelines, various
  reparameterization techniques have recently been introduced to sidestep the
  need for separately learning an RL reward model. Instead, directly fine-tuning
  for human preferences is achieved via the minimization of a single closed-form
  training objective, a process originally referred to as direct preference
  optimization (DPO) and followed by several notable descendants. Although
  effective in certain real-world settings, we introduce new evaluation criteria
  that serve to highlight unresolved shortcomings in the ability of existing DPO
  methods to interpolate between a pre-trained reference model and empirical
  measures of human preferences, as well as unavoidable trade-offs in how low-
  and high-quality responses are regularized and constraints are handled. Our
  insights then motivate an alternative DPO-like loss that provably mitigates
  these limitations. Empirical results serve to corroborate notable aspects of
  our analyses.
paper_authors._id: 6694d14ccab4aab691fa1c71
paper_authors.name: Xiangkun Hu
paper_authors._id: 6694d14ccab4aab691fa1c72
paper_authors.name: Tong He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-15T08:58:03.349Z
paper_authors._id: 63184606ab1bf092d6f059be
paper_authors.avatarUrl: /avatars/b49d125d112a01106b48a2fcda481c3e.svg
paper_authors.fullname: Tong He
paper_authors.type: user
paper_authors.user: hetong007
paper_authors._id: 6694d14ccab4aab691fa1c73
paper_authors.name: David Wipf
paper_title: New Desiderata for Direct Preference Optimization
paper_summary: Large language models in the past have typically relied on some form of
  reinforcement learning with human feedback (RLHF) to better align model
  responses with human preferences. However, because of oft-observed
  instabilities when implementing these RLHF pipelines, various
  reparameterization techniques have recently been introduced to sidestep the
  need for separately learning an RL reward model. Instead, directly fine-tuning
  for human preferences is achieved via the minimization of a single closed-form
  training objective, a process originally referred to as direct preference
  optimization (DPO) and followed by several notable descendants. Although
  effective in certain real-world settings, we introduce new evaluation criteria
  that serve to highlight unresolved shortcomings in the ability of existing DPO
  methods to interpolate between a pre-trained reference model and empirical
  measures of human preferences, as well as unavoidable trade-offs in how low-
  and high-quality responses are regularized and constraints are handled. Our
  insights then motivate an alternative DPO-like loss that provably mitigates
  these limitations. Empirical results serve to corroborate notable aspects of
  our analyses.
paper_ai_keywords: reinforcement learning with human feedback (RLHF) direct preference optimization (DPO) live video processing reparameterization pre-trained model human preferences training objective regularization constraints