Self-Rewarding Language Models
We posit that to achieve superhuman agents, future models require superhuman
feedback in order to provide an adequate training signal. Current approaches
commonly train reward models from human preferences, which may then be
bottlenecked by human performance level, and secondly these separate frozen
reward models cannot then learn to improve during LLM training. In this work,
we study Self-Rewarding Language Models, where the language model itself is
used via LLM-as-a-Judge prompting to provide its own rewards during training.
We show that during Iterative DPO training that not only does instruction
following ability improve, but also the ability to provide high-quality rewards
to itself. Fine-tuning Llama 2 70B on three iterations of our approach yields a
model that outperforms many existing systems on the AlpacaEval 2.0 leaderboard,
including Claude 2, Gemini Pro, and GPT-4 0613. While only a preliminary study,
this work opens the door to the possibility of models that can continually
improve in both axes.
65a9da1af2878c013d172f3c
Weizhe Yuan
admin_assigned
2024-01-19T09:25:20.277Z
6172aa78c8e66e2aa84c06b8
https://cdn-avatars.huggingface.co/v1/production/uploads/1634904686602-noauth.png
Weizhe Yuan
user
weizhey
65a9da1af2878c013d172f3d
Richard Yuanzhe Pang
admin_assigned
2024-01-19T09:26:44.842Z
64d50fa59a6a7ae984d90852
/avatars/26b78c151e707e3f9a9516c877d18ff7.svg
Richard Pang
user
yzpang
65a9da1af2878c013d172f3e
Kyunghyun Cho
admin_assigned
2024-01-19T09:26:52.380Z
649c1b8485b05ee02723334a
/avatars/29080f550fdb471c4d2daace86aff3b0.svg
Kyunghyun Cho
user
kyunghyuncho
65a9da1af2878c013d172f3f
Sainbayar Sukhbaatar
claimed_verified
2024-07-30T16:01:27.844Z
66a8611eb51510d82ed54231
/avatars/ad559e774fee4914091b82c9831ae2a2.svg
Sainbayar Sukhbaatar
user
sainbar
65a9da1af2878c013d172f40
Jing Xu
65a9da1af2878c013d172f41
Jason Weston
admin_assigned
2024-01-19T09:32:48.666Z
62f023a36a027498eaa2f9cc
/avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
Jason Weston
user
spermwhale
Self-Rewarding Language Models
We posit that to achieve superhuman agents, future models require superhuman
feedback in order to provide an adequate training signal. Current approaches
commonly train reward models from human preferences, which may then be
bottlenecked by human performance level, and secondly these separate frozen
reward models cannot then learn to improve during LLM training. In this work,
we study Self-Rewarding Language Models, where the language model itself is
used via LLM-as-a-Judge prompting to provide its own rewards during training.
We show that during Iterative DPO training that not only does instruction
following ability improve, but also the ability to provide high-quality rewards
to itself. Fine-tuning Llama 2 70B on three iterations of our approach yields a
model that outperforms many existing systems on the AlpacaEval 2.0 leaderboard,
including Claude 2, Gemini Pro, and GPT-4 0613. While only a preliminary study,
this work opens the door to the possibility of models that can continually
improve in both axes.
A study on Self-Rewarding Language Models shows that using LLM-as-a-Judge prompting for iterative DPO training enhances both instruction-following and self-reward generation, leading to superior performance compared to existing systems.
Self-Rewarding Language Models LLM-as-a-Judge Iterative DPO training AlpacaEval 2.0 Claude 2 Gemini Pro GPT-4 0613