Self-Taught Evaluators
Model-based evaluation is at the heart of successful model development -- as
a reward model for training, and as a replacement for human evaluation. To
train such evaluators, the standard approach is to collect a large amount of
human preference judgments over model responses, which is costly and the data
becomes stale as models improve. In this work, we present an approach that aims
to im-prove evaluators without human annotations, using synthetic training data
only. Starting from unlabeled instructions, our iterative self-improvement
scheme generates contrasting model outputs and trains an LLM-as-a-Judge to
produce reasoning traces and final judgments, repeating this training at each
new iteration using the improved predictions. Without any labeled preference
data, our Self-Taught Evaluator can improve a strong LLM (Llama3-70B-Instruct)
from 75.4 to 88.3 (88.7 with majority vote) on RewardBench. This outperforms
commonly used LLM judges such as GPT-4 and matches the performance of the
top-performing reward models trained with labeled examples.
66b18cfc973a754d682276cf
Tianlu Wang
admin_assigned
2024-08-06T10:34:59.786Z
63f6a0e6b4c9a104f4b95d86
/avatars/660c3e44bea976f3e0560bf95466b03c.svg
Tianlu Wang
user
Tianlu
66b18cfc973a754d682276d0
Ilia Kulikov
admin_assigned
2024-08-06T10:35:05.124Z
65d645913bc37244c55c39d5
/avatars/508372876a654957650ee9429886d10c.svg
ilia kulikov
user
uralik
66b18cfc973a754d682276d1
Olga Golovneva
admin_assigned
2024-08-06T10:35:10.406Z
6318b1e06e6bd039440871d6
/avatars/2d9f96eb4092a9d1734dcc09303b259a.svg
Olga Golovneva
user
Golovneva
66b18cfc973a754d682276d2
Ping Yu
admin_assigned
2024-08-06T10:35:22.665Z
646c004e31968a60a022590e
/avatars/349e2ed4feef1be050fd9903dccec1e1.svg
Ping Yu
user
pingyu
66b18cfc973a754d682276d3
Weizhe Yuan
admin_assigned
2024-08-06T10:35:28.021Z
6172aa78c8e66e2aa84c06b8
https://cdn-avatars.huggingface.co/v1/production/uploads/1634904686602-noauth.png
Weizhe Yuan
user
weizhey
66b18cfc973a754d682276d4
Jane Dwivedi-Yu
66b18cfc973a754d682276d5
Richard Yuanzhe Pang
admin_assigned
2024-08-06T10:36:05.594Z
64d50fa59a6a7ae984d90852
/avatars/26b78c151e707e3f9a9516c877d18ff7.svg
Richard Pang
user
yzpang
66b18cfc973a754d682276d6
Maryam Fazel-Zarandi
66b18cfc973a754d682276d7
Jason Weston
admin_assigned
2024-08-06T10:36:29.164Z
62f023a36a027498eaa2f9cc
/avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
Jason Weston
user
spermwhale
66b18cfc973a754d682276d8
Xian Li
Self-Taught Evaluators
Model-based evaluation is at the heart of successful model development -- as
a reward model for training, and as a replacement for human evaluation. To
train such evaluators, the standard approach is to collect a large amount of
human preference judgments over model responses, which is costly and the data
becomes stale as models improve. In this work, we present an approach that aims
to im-prove evaluators without human annotations, using synthetic training data
only. Starting from unlabeled instructions, our iterative self-improvement
scheme generates contrasting model outputs and trains an LLM-as-a-Judge to
produce reasoning traces and final judgments, repeating this training at each
new iteration using the improved predictions. Without any labeled preference
data, our Self-Taught Evaluator can improve a strong LLM (Llama3-70B-Instruct)
from 75.4 to 88.3 (88.7 with majority vote) on RewardBench. This outperforms
commonly used LLM judges such as GPT-4 and matches the performance of the
top-performing reward models trained with labeled examples.
A self-improving evaluator trained with synthetic data outperforms traditional human-annotated evaluators and matches top-performing reward models on RewardBench.
LLM-as-a-Judge reasoning traces iterative self-improvement synthetic training data RewardBench Llama3-70B-Instruct GPT-4