title: Learn Your Reference Model for Real Good Alignment
summary: The complexity of the alignment problem stems from the fact that existing
  methods are unstable. Researchers continuously invent various tricks to address
  this shortcoming. For instance, in the fundamental Reinforcement Learning From
  Human Feedback (RLHF) technique of Language Model alignment, in addition to
  reward maximization, the Kullback-Leibler divergence between the trainable
  policy and the SFT policy is minimized. This addition prevents the model from
  being overfitted to the Reward Model (RM) and generating texts that are
  out-of-domain for the RM. The Direct Preference Optimization (DPO) method
  reformulates the optimization task of RLHF and eliminates the Reward Model
  while tacitly maintaining the requirement for the policy to be close to the SFT
  policy. In our paper, we argue that this implicit limitation in the DPO method
  leads to sub-optimal results. We propose a new method called Trust Region DPO
  (TR-DPO), which updates the reference policy during training. With such a
  straightforward update, we demonstrate the effectiveness of TR-DPO against DPO
  on the Anthropic HH and TLDR datasets. We show that TR-DPO outperforms DPO by
  up to 19%, measured by automatic evaluation with GPT-4. The new alignment
  approach that we propose allows us to improve the quality of models across
  several parameters at once, such as coherence, correctness, level of detail,
  helpfulness, and harmlessness.
paper_authors._id: 661de9b836eb62686fcce152
paper_authors.name: Alexey Gorbatovski
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-16T08:59:53.613Z
paper_authors._id: 62897fce5d9e25c10e4f319d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62897fce5d9e25c10e4f319d/bMlfAyzkNNZlkQ5mCW6Vc.jpeg
paper_authors.fullname: Alexey Gorbatovski
paper_authors.type: user
paper_authors.user: Myashka
paper_authors._id: 661de9b836eb62686fcce153
paper_authors.name: Boris Shaposhnikov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-16T09:04:54.014Z
paper_authors._id: 637dd11dcbad6e62a5e39743
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637dd11dcbad6e62a5e39743/DG3rM8cy8inqbCoG4qizO.jpeg
paper_authors.fullname: Boris Shaposhnikov
paper_authors.type: user
paper_authors.user: borisshapa
paper_authors._id: 661de9b836eb62686fcce154
paper_authors.name: Alexey Malakhov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-16T10:43:31.078Z
paper_authors._id: 636e71b2b0ebc04888157b71
paper_authors.avatarUrl: /avatars/957ba705d470e3a01792741d7f0ff038.svg
paper_authors.fullname: Alexey Malakhov
paper_authors.type: user
paper_authors.user: ZeL1k7
paper_authors._id: 661de9b836eb62686fcce155
paper_authors.name: Nikita Surnachev
paper_authors._id: 661de9b836eb62686fcce156
paper_authors.name: Yaroslav Aksenov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-16T23:16:54.902Z
paper_authors._id: 63ed5676684767daecac6f8a
paper_authors.avatarUrl: /avatars/d0e4a715f9c3fb6d74c183bab751ec35.svg
paper_authors.fullname: Yaroslav Aksenov
paper_authors.type: user
paper_authors.user: yaraksen
paper_authors._id: 661de9b836eb62686fcce157
paper_authors.name: Ian Maksimov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-16T12:44:33.851Z
paper_authors._id: 65cbcb760b3bd8f5ce402bd5
paper_authors.avatarUrl: /avatars/2a60e344cf219eed58217ccf5db26680.svg
paper_authors.fullname: Ian Maksimov
paper_authors.type: user
paper_authors.user: BayesianMonster
paper_authors._id: 661de9b836eb62686fcce158
paper_authors.name: Nikita Balagansky
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-16T08:20:16.276Z
paper_authors._id: 60b364e7f88532cd79eaff7b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654185363389-60b364e7f88532cd79eaff7b.jpeg
paper_authors.fullname: Nikita Balagansky
paper_authors.type: user
paper_authors.user: elephantmipt
paper_authors._id: 661de9b836eb62686fcce159
paper_authors.name: Daniil Gavrilov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-16T08:15:54.509Z
paper_authors._id: 62a9c8edc19f92ae443ab37f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669110208492-62a9c8edc19f92ae443ab37f.png
paper_authors.fullname: Daniil Gavrilov
paper_authors.type: user
paper_authors.user: kefirski
paper_title: Learn Your Reference Model for Real Good Alignment
paper_summary: The complexity of the alignment problem stems from the fact that existing
  methods are unstable. Researchers continuously invent various tricks to address
  this shortcoming. For instance, in the fundamental Reinforcement Learning From
  Human Feedback (RLHF) technique of Language Model alignment, in addition to
  reward maximization, the Kullback-Leibler divergence between the trainable
  policy and the SFT policy is minimized. This addition prevents the model from
  being overfitted to the Reward Model (RM) and generating texts that are
  out-of-domain for the RM. The Direct Preference Optimization (DPO) method
  reformulates the optimization task of RLHF and eliminates the Reward Model
  while tacitly maintaining the requirement for the policy to be close to the SFT
  policy. In our paper, we argue that this implicit limitation in the DPO method
  leads to sub-optimal results. We propose a new method called Trust Region DPO
  (TR-DPO), which updates the reference policy during training. With such a
  straightforward update, we demonstrate the effectiveness of TR-DPO against DPO
  on the Anthropic HH and TLDR datasets. We show that TR-DPO outperforms DPO by
  up to 19%, measured by automatic evaluation with GPT-4. The new alignment
  approach that we propose allows us to improve the quality of models across
  several parameters at once, such as coherence, correctness, level of detail,
  helpfulness, and harmlessness.
paper_ai_keywords: Reinforcement Learning From Human Feedback (RLHF) Kullback-Leibler divergence trainable policy SFT policy Reward Model (RMModel) overfitted out-of-domain Direct Preference Optimization (DPO) Trust Region DPO (TR-DPO) reference policy Anthropic HH dataset TLDR dataset GPT-4 coherence correctness level of detail helpfulness harmlessness