title: Meta-Rewarding Language Models: Self-Improving Alignment with
    LLM-as-a-Meta-Judge
summary: Large Language Models (LLMs) are rapidly surpassing human knowledge in many
  domains. While improving these models traditionally relies on costly human
  data, recent self-rewarding mechanisms (Yuan et al., 2024) have shown that LLMs
  can improve by judging their own responses instead of relying on human
  labelers. However, existing methods have primarily focused on improving model
  responses rather than judgment capabilities, resulting in rapid saturation
  during iterative training. To address this issue, we introduce a novel
  Meta-Rewarding step to the self-improvement process, where the model judges its
  own judgements and uses that feedback to refine its judgment skills.
  Surprisingly, this unsupervised approach improves the model's ability to judge
  {\em and} follow instructions, as demonstrated by a win rate improvement of
  Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2, and 20.6% to 29.1% on
  Arena-Hard. These results strongly suggest the potential for self-improving
  models without human supervision.
paper_authors._id: 66a90d9fbaa2e83d93741531
paper_authors.name: Tianhao Wu
paper_authors._id: 66a90d9fbaa2e83d93741532
paper_authors.name: Weizhe Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-31T07:44:41.503Z
paper_authors._id: 6172aa78c8e66e2aa84c06b8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1634904686602-noauth.png
paper_authors.fullname: Weizhe Yuan
paper_authors.type: user
paper_authors.user: weizhey
paper_authors._id: 66a90d9fbaa2e83d93741533
paper_authors.name: Olga Golovneva
paper_authors._id: 66a90d9fbaa2e83d93741534
paper_authors.name: Jing Xu
paper_authors._id: 66a90d9fbaa2e83d93741535
paper_authors.name: Yuandong Tian
paper_authors._id: 66a90d9fbaa2e83d93741536
paper_authors.name: Jiantao Jiao
paper_authors._id: 66a90d9fbaa2e83d93741537
paper_authors.name: Jason Weston
paper_authors._id: 66a90d9fbaa2e83d93741538
paper_authors.name: Sainbayar Sukhbaatar
paper_title: Meta-Rewarding Language Models: Self-Improving Alignment with
    LLM-as-a-Meta-Judge
paper_summary: Large Language Models (LLMs) are rapidly surpassing human knowledge in many
  domains. While improving these models traditionally relies on costly human
  data, recent self-rewarding mechanisms (Yuan et al., 2024) have shown that LLMs
  can improve by judging their own responses instead of relying on human
  labelers. However, existing methods have primarily focused on improving model
  responses rather than judgment capabilities, resulting in rapid saturation
  during iterative training. To address this issue, we introduce a novel
  Meta-Rewarding step to the self-improvement process, where the model judges its
  own judgements and uses that feedback to refine its judgment skills.
  Surprisingly, this unsupervised approach improves the model's ability to judge
  {\em and} follow instructions, as demonstrated by a win rate improvement of
  Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2, and 20.6% to 29.1% on
  Arena-Hard. These results strongly suggest the potential for self-improving
  models without human supervision.
paper_ai_keywords: Meta-Rewarding self-rewarding mechanisms Meta-Rewarding step self-improvement process iterative training judgment capabilities control to judge follow instructions AlpacaEval 2 Arena-Hard win rate improvement self-improving models human supervision