title: Statistical Rejection Sampling Improves Preference Optimization
summary: Improving the alignment of language models with human preferences remains an
  active research challenge. Previous approaches have primarily utilized
  Reinforcement Learning from Human Feedback (RLHF) via online RL methods such as
  Proximal Policy Optimization (PPO). Recently, offline methods such as Sequence
  Likelihood Calibration (SLiC) and Direct Preference Optimization (DPO) have
  emerged as attractive alternatives, offering improvements in stability and
  scalability while maintaining competitive performance. SLiC refines its loss
  function using sequence pairs sampled from a supervised fine-tuned (SFT)
  policy, while DPO directly optimizes language models based on preference data,
  foregoing the need for a separate reward model. However, the maximum likelihood
  estimator (MLE) of the target optimal policy requires labeled preference pairs
  sampled from that policy. DPO's lack of a reward model constrains its ability
  to sample preference pairs from the optimal policy, and SLiC is restricted to
  sampling preference pairs only from the SFT policy. To address these
  limitations, we introduce a novel approach called Statistical Rejection
  Sampling Optimization (RSO) that aims to source preference data from the target
  optimal policy using rejection sampling, enabling a more accurate estimation of
  the optimal policy. We also propose a unified framework that enhances the loss
  functions used in both SLiC and DPO from a preference modeling standpoint.
  Through extensive experiments across three diverse tasks, we demonstrate that
  RSO consistently outperforms both SLiC and DPO on evaluations from both Large
  Language Model (LLM) and human raters.
paper_authors._id: 65026cf6e751d03da934a16f
paper_authors.name: Tianqi Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-14T07:35:32.897Z
paper_authors._id: 646784af696e7355f5d5001f
paper_authors.avatarUrl: /avatars/39ce8999c296584ee0bdeb7848eee6d9.svg
paper_authors.fullname: Tianqi Liu
paper_authors.type: user
paper_authors.user: TianqiLiuAI
paper_authors._id: 65026cf6e751d03da934a170
paper_authors.name: Yao Zhao
paper_authors._id: 65026cf6e751d03da934a171
paper_authors.name: Rishabh Joshi
paper_authors._id: 65026cf6e751d03da934a172
paper_authors.name: Misha Khalman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-07T20:41:23.216Z
paper_authors._id: 65c2ad2790703d2dbcf79471
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65c2ad2790703d2dbcf79471/H15_ASSI7fCFDCDF4xgIb.jpeg
paper_authors.fullname: Misha Khalman
paper_authors.type: user
paper_authors.user: khalman
paper_authors._id: 65026cf6e751d03da934a173
paper_authors.name: Mohammad Saleh
paper_authors._id: 65026cf6e751d03da934a174
paper_authors.name: Peter J. Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-14T07:37:13.544Z
paper_authors._id: 62ded294dcdc9043efaa4fc3
paper_authors.avatarUrl: /avatars/1294078b8cd5c1901e4021253d877357.svg
paper_authors.fullname: Peter J Liu
paper_authors.type: user
paper_authors.user: peterjliu
paper_authors._id: 65026cf6e751d03da934a175
paper_authors.name: Jialu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-14T07:37:20.469Z
paper_authors._id: 64acb90a5d48838462da09ae
paper_authors.avatarUrl: /avatars/dc87416dcb2806a6e0d60bd636024978.svg
paper_authors.fullname: liujialu
paper_authors.type: user
paper_authors.user: Jialuliu
paper_title: Statistical Rejection Sampling Improves Preference Optimization
paper_summary: Improving the alignment of language models with human preferences remains an
  active research challenge. Previous approaches have primarily utilized
  Reinforcement Learning from Human Feedback (RLHF) via online RL methods such as
  Proximal Policy Optimization (PPO). Recently, offline methods such as Sequence
  Likelihood Calibration (SLiC) and Direct Preference Optimization (DPO) have
  emerged as attractive alternatives, offering improvements in stability and
  scalability while maintaining competitive performance. SLiC refines its loss
  function using sequence pairs sampled from a supervised fine-tuned (SFT)
  policy, while DPO directly optimizes language models based on preference data,
  foregoing the need for a separate reward model. However, the maximum likelihood
  estimator (MLE) of the target optimal policy requires labeled preference pairs
  sampled from that policy. DPO's lack of a reward model constrains its ability
  to sample preference pairs from the optimal policy, and SLiC is restricted to
  sampling preference pairs only from the SFT policy. To address these
  limitations, we introduce a novel approach called Statistical Rejection
  Sampling Optimization (RSO) that aims to source preference data from the target
  optimal policy using rejection sampling, enabling a more accurate estimation of
  the optimal policy. We also propose a unified framework that enhances the loss
  functions used in both SLiC and DPO from a preference modeling standpoint.
  Through extensive experiments across three diverse tasks, we demonstrate that
  RSO consistently outperforms both SLiC and DPO on evaluations from both Large
  Language Model (LLM) and human raters.
paper_ai_keywords: Reinforcement Learning from Human Feedback (RLHF) Proximal Policy Optimization (PPO) Sequence Likelihood Calibration (SLiC) Direct Preference Optimization (DPO) maximum likelihood estimator (MLE) Statistical Rejection Sampling Optimization (RSO) rejection sampling unified framework Large Language Model (LLM)