Bootstrapping Language Models with DPO Implicit Rewards
Human alignment in large language models (LLMs) is an active area of
research. A recent groundbreaking work, direct preference optimization (DPO),
has greatly simplified the process from past work in reinforcement learning
from human feedback (RLHF) by bypassing the reward learning stage in RLHF. DPO,
after training, provides an implicit reward model. In this work, we make a
novel observation that this implicit reward model can by itself be used in a
bootstrapping fashion to further align the LLM. Our approach is to use the
rewards from a current LLM model to construct a preference dataset, which is
then used in subsequent DPO rounds. We incorporate refinements that debias the
length of the responses and improve the quality of the preference dataset to
further improve our approach. Our approach, named self-alignment with DPO
ImpliCit rEwards (DICE), shows great improvements in alignment and achieves
superior performance than Gemini Pro on AlpacaEval 2, reaching 27.55%
length-controlled win rate against GPT-4 Turbo, but with only 8B parameters and
no external feedback. Our code is available at https://github.com/sail-sg/dice.
66727d57a8a27a82f40dc47b
Changyu Chen
claimed_verified
2024-06-19T20:20:42.676Z
64e416dc54e18f390ef79ba4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/5n01J00ZaVRrebsON8iYA.jpeg
Changyu Chen
user
Cameron-Chen
66727d57a8a27a82f40dc47c
Zichen Liu
claimed_verified
2024-11-15T09:46:52.337Z
65f5392c68b8e0cb3c9977a2
/avatars/aa64772475098e8a135c13072fde6744.svg
Zichen
user
lkevinzc
66727d57a8a27a82f40dc47d
Chao Du
claimed_verified
2024-06-19T20:20:44.823Z
632407c892e07e3ca20aca28
/avatars/23b51b37b12b51a0947f687d1de4d3b5.svg
Chao Du
user
duchao
66727d57a8a27a82f40dc47e
Tianyu Pang
admin_assigned
2024-06-19T08:36:46.369Z
63d91b6d255ef6add20e1b38
https://cdn-avatars.huggingface.co/v1/production/uploads/1675921369867-63d91b6d255ef6add20e1b38.jpeg
Tianyu Pang
user
P2333
66727d57a8a27a82f40dc47f
Qian Liu
claimed_verified
2024-06-19T07:37:22.994Z
612ee6a7b960e78c6d2319d4
https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
Qian Liu
user
SivilTaram
66727d57a8a27a82f40dc480
Arunesh Sinha
66727d57a8a27a82f40dc481
Pradeep Varakantham
admin_assigned
2024-06-19T08:36:55.687Z
639a424b21b7435ae152f134
/avatars/404d682fea0dafa9267c00d067b5b26c.svg
Pradeep varakantham
user
Pradeepv
66727d57a8a27a82f40dc482
Min Lin
Bootstrapping Language Models with DPO Implicit Rewards
Human alignment in large language models (LLMs) is an active area of
research. A recent groundbreaking work, direct preference optimization (DPO),
has greatly simplified the process from past work in reinforcement learning
from human feedback (RLHF) by bypassing the reward learning stage in RLHF. DPO,
after training, provides an implicit reward model. In this work, we make a
novel observation that this implicit reward model can by itself be used in a
bootstrapping fashion to further align the LLM. Our approach is to use the
rewards from a current LLM model to construct a preference dataset, which is
then used in subsequent DPO rounds. We incorporate refinements that debias the
length of the responses and improve the quality of the preference dataset to
further improve our approach. Our approach, named self-alignment with DPO
ImpliCit rEwards (DICE), shows great improvements in alignment and achieves
superior performance than Gemini Pro on AlpacaEval 2, reaching 27.55%
length-controlled win rate against GPT-4 Turbo, but with only 8B parameters and
no external feedback. Our code is available at https://github.com/sail-sg/dice.
A novel method using the implicit reward model from Direct Preference Optimization (DPO) to iteratively improve the alignment of large language models, achieving superior performance without external feedback.
direct preference optimization (DPO) reinforcement learning from human feedback (RLHF) implicit reward model preference dataset bootstrapping self-alignment AlpacaEval 2 GPT-4 Turbo