title: Bootstrapping Language Models with DPO Implicit Rewards
summary: Human alignment in large language models (LLMs) is an active area of
  research. A recent groundbreaking work, direct preference optimization (DPO),
  has greatly simplified the process from past work in reinforcement learning
  from human feedback (RLHF) by bypassing the reward learning stage in RLHF. DPO,
  after training, provides an implicit reward model. In this work, we make a
  novel observation that this implicit reward model can by itself be used in a
  bootstrapping fashion to further align the LLM. Our approach is to use the
  rewards from a current LLM model to construct a preference dataset, which is
  then used in subsequent DPO rounds. We incorporate refinements that debias the
  length of the responses and improve the quality of the preference dataset to
  further improve our approach. Our approach, named self-alignment with DPO
  ImpliCit rEwards (DICE), shows great improvements in alignment and achieves
  superior performance than Gemini Pro on AlpacaEval 2, reaching 27.55%
  length-controlled win rate against GPT-4 Turbo, but with only 8B parameters and
  no external feedback. Our code is available at https://github.com/sail-sg/dice.
paper_authors._id: 66727d57a8a27a82f40dc47b
paper_authors.name: Changyu Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T20:20:42.676Z
paper_authors._id: 64e416dc54e18f390ef79ba4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/5n01J00ZaVRrebsON8iYA.jpeg
paper_authors.fullname: Changyu Chen
paper_authors.type: user
paper_authors.user: Cameron-Chen
paper_authors._id: 66727d57a8a27a82f40dc47c
paper_authors.name: Zichen Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:46:52.337Z
paper_authors._id: 65f5392c68b8e0cb3c9977a2
paper_authors.avatarUrl: /avatars/aa64772475098e8a135c13072fde6744.svg
paper_authors.fullname: Zichen
paper_authors.type: user
paper_authors.user: lkevinzc
paper_authors._id: 66727d57a8a27a82f40dc47d
paper_authors.name: Chao Du
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T20:20:44.823Z
paper_authors._id: 632407c892e07e3ca20aca28
paper_authors.avatarUrl: /avatars/23b51b37b12b51a0947f687d1de4d3b5.svg
paper_authors.fullname: Chao Du
paper_authors.type: user
paper_authors.user: duchao
paper_authors._id: 66727d57a8a27a82f40dc47e
paper_authors.name: Tianyu Pang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T08:36:46.369Z
paper_authors._id: 63d91b6d255ef6add20e1b38
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1675921369867-63d91b6d255ef6add20e1b38.jpeg
paper_authors.fullname: Tianyu Pang
paper_authors.type: user
paper_authors.user: P2333
paper_authors._id: 66727d57a8a27a82f40dc47f
paper_authors.name: Qian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T07:37:22.994Z
paper_authors._id: 612ee6a7b960e78c6d2319d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
paper_authors.fullname: Qian Liu
paper_authors.type: user
paper_authors.user: SivilTaram
paper_authors._id: 66727d57a8a27a82f40dc480
paper_authors.name: Arunesh Sinha
paper_authors._id: 66727d57a8a27a82f40dc481
paper_authors.name: Pradeep Varakantham
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T08:36:55.687Z
paper_authors._id: 639a424b21b7435ae152f134
paper_authors.avatarUrl: /avatars/404d682fea0dafa9267c00d067b5b26c.svg
paper_authors.fullname: Pradeep varakantham
paper_authors.type: user
paper_authors.user: Pradeepv
paper_authors._id: 66727d57a8a27a82f40dc482
paper_authors.name: Min Lin
paper_title: Bootstrapping Language Models with DPO Implicit Rewards
paper_summary: Human alignment in large language models (LLMs) is an active area of
  research. A recent groundbreaking work, direct preference optimization (DPO),
  has greatly simplified the process from past work in reinforcement learning
  from human feedback (RLHF) by bypassing the reward learning stage in RLHF. DPO,
  after training, provides an implicit reward model. In this work, we make a
  novel observation that this implicit reward model can by itself be used in a
  bootstrapping fashion to further align the LLM. Our approach is to use the
  rewards from a current LLM model to construct a preference dataset, which is
  then used in subsequent DPO rounds. We incorporate refinements that debias the
  length of the responses and improve the quality of the preference dataset to
  further improve our approach. Our approach, named self-alignment with DPO
  ImpliCit rEwards (DICE), shows great improvements in alignment and achieves
  superior performance than Gemini Pro on AlpacaEval 2, reaching 27.55%
  length-controlled win rate against GPT-4 Turbo, but with only 8B parameters and
  no external feedback. Our code is available at https://github.com/sail-sg/dice.
paper_ai_keywords: direct preference optimization (DPO) reinforcement learning from human feedback (RLHF) reward model preference dataset self-alignment with DPO implementation rewards (DICE) AlpacaEval length-controlled win rate GPT-4 Turbo