title: Self-Rewarding Language Models
summary: We posit that to achieve superhuman agents, future models require superhuman
  feedback in order to provide an adequate training signal. Current approaches
  commonly train reward models from human preferences, which may then be
  bottlenecked by human performance level, and secondly these separate frozen
  reward models cannot then learn to improve during LLM training. In this work,
  we study Self-Rewarding Language Models, where the language model itself is
  used via LLM-as-a-Judge prompting to provide its own rewards during training.
  We show that during Iterative DPO training that not only does instruction
  following ability improve, but also the ability to provide high-quality rewards
  to itself. Fine-tuning Llama 2 70B on three iterations of our approach yields a
  model that outperforms many existing systems on the AlpacaEval 2.0 leaderboard,
  including Claude 2, Gemini Pro, and GPT-4 0613. While only a preliminary study,
  this work opens the door to the possibility of models that can continually
  improve in both axes.
paper_authors._id: 65a9da1af2878c013d172f3c
paper_authors.name: Weizhe Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-19T09:25:20.277Z
paper_authors._id: 6172aa78c8e66e2aa84c06b8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1634904686602-noauth.png
paper_authors.fullname: Weizhe Yuan
paper_authors.type: user
paper_authors.user: weizhey
paper_authors._id: 65a9da1af2878c013d172f3d
paper_authors.name: Richard Yuanzhe Pang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-19T09:26:44.842Z
paper_authors._id: 64d50fa59a6a7ae984d90852
paper_authors.avatarUrl: /avatars/26b78c151e707e3f9a9516c877d18ff7.svg
paper_authors.fullname: Richard Pang
paper_authors.type: user
paper_authors.user: yzpang
paper_authors._id: 65a9da1af2878c013d172f3e
paper_authors.name: Kyunghyun Cho
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-19T09:26:52.380Z
paper_authors._id: 649c1b8485b05ee02723334a
paper_authors.avatarUrl: /avatars/29080f550fdb471c4d2daace86aff3b0.svg
paper_authors.fullname: Kyunghyun Cho
paper_authors.type: user
paper_authors.user: kyunghyuncho
paper_authors._id: 65a9da1af2878c013d172f3f
paper_authors.name: Sainbayar Sukhbaatar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-30T16:01:27.844Z
paper_authors._id: 66a8611eb51510d82ed54231
paper_authors.avatarUrl: /avatars/ad559e774fee4914091b82c9831ae2a2.svg
paper_authors.fullname: Sainbayar Sukhbaatar
paper_authors.type: user
paper_authors.user: sainbar
paper_authors._id: 65a9da1af2878c013d172f40
paper_authors.name: Jing Xu
paper_authors._id: 65a9da1af2878c013d172f41
paper_authors.name: Jason Weston
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-19T09:32:48.666Z
paper_authors._id: 62f023a36a027498eaa2f9cc
paper_authors.avatarUrl: /avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
paper_authors.fullname: Jason Weston
paper_authors.type: user
paper_authors.user: spermwhale
paper_title: Self-Rewarding Language Models
paper_summary: We posit that to achieve superhuman agents, future models require superhuman
  feedback in order to provide an adequate training signal. Current approaches
  commonly train reward models from human preferences, which may then be
  bottlenecked by human performance level, and secondly these separate frozen
  reward models cannot then learn to improve during LLM training. In this work,
  we study Self-Rewarding Language Models, where the language model itself is
  used via LLM-as-a-Judge prompting to provide its own rewards during training.
  We show that during Iterative DPO training that not only does instruction
  following ability improve, but also the ability to provide high-quality rewards
  to itself. Fine-tuning Llama 2 70B on three iterations of our approach yields a
  model that outperforms many existing systems on the AlpacaEval 2.0 leaderboard,
  including Claude 2, Gemini Pro, and GPT-4 0613. While only a preliminary study,
  this work opens the door to the possibility of models that can continually
  improve in both axes.
paper_ai_keywords: Self-Rewarding Language Models LLM-as-a-Judge prompting Iterative DPO training instruction following ability high-quality rewards Fine-tuning Llama 2 70B AlpacaEval 2.0 leaderboard Claude 2 Gemini Pro GPT-4 0613