Meta-Rewarding Language Models: Self-Improving Alignment with
  LLM-as-a-Meta-Judge
Large Language Models (LLMs) are rapidly surpassing human knowledge in many
domains. While improving these models traditionally relies on costly human
data, recent self-rewarding mechanisms (Yuan et al., 2024) have shown that LLMs
can improve by judging their own responses instead of relying on human
labelers. However, existing methods have primarily focused on improving model
responses rather than judgment capabilities, resulting in rapid saturation
during iterative training. To address this issue, we introduce a novel
Meta-Rewarding step to the self-improvement process, where the model judges its
own judgements and uses that feedback to refine its judgment skills.
Surprisingly, this unsupervised approach improves the model's ability to judge
{\em and} follow instructions, as demonstrated by a win rate improvement of
Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2, and 20.6% to 29.1% on
Arena-Hard. These results strongly suggest the potential for self-improving
models without human supervision.
66a90d9fbaa2e83d93741531
Tianhao Wu
66a90d9fbaa2e83d93741532
Weizhe Yuan
claimed_verified
2024-07-31T07:44:41.503Z
6172aa78c8e66e2aa84c06b8
https://cdn-avatars.huggingface.co/v1/production/uploads/1634904686602-noauth.png
Weizhe Yuan
user
weizhey
66a90d9fbaa2e83d93741533
Olga Golovneva
66a90d9fbaa2e83d93741534
Jing Xu
66a90d9fbaa2e83d93741535
Yuandong Tian
66a90d9fbaa2e83d93741536
Jiantao Jiao
66a90d9fbaa2e83d93741537
Jason Weston
66a90d9fbaa2e83d93741538
Sainbayar Sukhbaatar
claimed_verified
2025-06-05T07:31:01.807Z
66a8611eb51510d82ed54231
/avatars/ad559e774fee4914091b82c9831ae2a2.svg
Sainbayar Sukhbaatar
user
sainbar
Meta-Rewarding Language Models: Self-Improving Alignment with
  LLM-as-a-Meta-Judge
Large Language Models (LLMs) are rapidly surpassing human knowledge in many
domains. While improving these models traditionally relies on costly human
data, recent self-rewarding mechanisms (Yuan et al., 2024) have shown that LLMs
can improve by judging their own responses instead of relying on human
labelers. However, existing methods have primarily focused on improving model
responses rather than judgment capabilities, resulting in rapid saturation
during iterative training. To address this issue, we introduce a novel
Meta-Rewarding step to the self-improvement process, where the model judges its
own judgements and uses that feedback to refine its judgment skills.
Surprisingly, this unsupervised approach improves the model's ability to judge
{\em and} follow instructions, as demonstrated by a win rate improvement of
Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2, and 20.6% to 29.1% on
Arena-Hard. These results strongly suggest the potential for self-improving
models without human supervision.
A novel Meta-Rewarding step in self-improvement enhances LLM judgment capabilities and instruction-following ability without human supervision.
Large Language Models self-rewarding mechanisms Meta-Rewarding AlpacaEval Arena-Hard