title: Self-Taught Evaluators
summary: Model-based evaluation is at the heart of successful model development -- as
  a reward model for training, and as a replacement for human evaluation. To
  train such evaluators, the standard approach is to collect a large amount of
  human preference judgments over model responses, which is costly and the data
  becomes stale as models improve. In this work, we present an approach that aims
  to im-prove evaluators without human annotations, using synthetic training data
  only. Starting from unlabeled instructions, our iterative self-improvement
  scheme generates contrasting model outputs and trains an LLM-as-a-Judge to
  produce reasoning traces and final judgments, repeating this training at each
  new iteration using the improved predictions. Without any labeled preference
  data, our Self-Taught Evaluator can improve a strong LLM (Llama3-70B-Instruct)
  from 75.4 to 88.3 (88.7 with majority vote) on RewardBench. This outperforms
  commonly used LLM judges such as GPT-4 and matches the performance of the
  top-performing reward models trained with labeled examples.
paper_authors._id: 66b18cfc973a754d682276cf
paper_authors.name: Tianlu Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T10:34:59.786Z
paper_authors._id: 63f6a0e6b4c9a104f4b95d86
paper_authors.avatarUrl: /avatars/660c3e44bea976f3e0560bf95466b03c.svg
paper_authors.fullname: Tianlu Wang
paper_authors.type: user
paper_authors.user: Tianlu
paper_authors._id: 66b18cfc973a754d682276d0
paper_authors.name: Ilia Kulikov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T10:35:05.124Z
paper_authors._id: 65d645913bc37244c55c39d5
paper_authors.avatarUrl: /avatars/508372876a654957650ee9429886d10c.svg
paper_authors.fullname: ilia kulikov
paper_authors.type: user
paper_authors.user: uralik
paper_authors._id: 66b18cfc973a754d682276d1
paper_authors.name: Olga Golovneva
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T10:35:10.406Z
paper_authors._id: 6318b1e06e6bd039440871d6
paper_authors.avatarUrl: /avatars/2d9f96eb4092a9d1734dcc09303b259a.svg
paper_authors.fullname: Olga Golovneva
paper_authors.type: user
paper_authors.user: Golovneva
paper_authors._id: 66b18cfc973a754d682276d2
paper_authors.name: Ping Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T10:35:22.665Z
paper_authors._id: 646c004e31968a60a022590e
paper_authors.avatarUrl: /avatars/349e2ed4feef1be050fd9903dccec1e1.svg
paper_authors.fullname: Ping Yu
paper_authors.type: user
paper_authors.user: pingyu
paper_authors._id: 66b18cfc973a754d682276d3
paper_authors.name: Weizhe Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T10:35:28.021Z
paper_authors._id: 6172aa78c8e66e2aa84c06b8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1634904686602-noauth.png
paper_authors.fullname: Weizhe Yuan
paper_authors.type: user
paper_authors.user: weizhey
paper_authors._id: 66b18cfc973a754d682276d4
paper_authors.name: Jane Dwivedi-Yu
paper_authors._id: 66b18cfc973a754d682276d5
paper_authors.name: Richard Yuanzhe Pang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T10:36:05.594Z
paper_authors._id: 64d50fa59a6a7ae984d90852
paper_authors.avatarUrl: /avatars/26b78c151e707e3f9a9516c877d18ff7.svg
paper_authors.fullname: Richard Pang
paper_authors.type: user
paper_authors.user: yzpang
paper_authors._id: 66b18cfc973a754d682276d6
paper_authors.name: Maryam Fazel-Zarandi
paper_authors._id: 66b18cfc973a754d682276d7
paper_authors.name: Jason Weston
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T10:36:29.164Z
paper_authors._id: 62f023a36a027498eaa2f9cc
paper_authors.avatarUrl: /avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
paper_authors.fullname: Jason Weston
paper_authors.type: user
paper_authors.user: spermwhale
paper_authors._id: 66b18cfc973a754d682276d8
paper_authors.name: Xian Li
paper_title: Self-Taught Evaluators
paper_summary: Model-based evaluation is at the heart of successful model development -- as
  a reward model for training, and as a replacement for human evaluation. To
  train such evaluators, the standard approach is to collect a large amount of
  human preference judgments over model responses, which is costly and the data
  becomes stale as models improve. In this work, we present an approach that aims
  to im-prove evaluators without human annotations, using synthetic training data
  only. Starting from unlabeled instructions, our iterative self-improvement
  scheme generates contrasting model outputs and trains an LLM-as-a-Judge to
  produce reasoning traces and final judgments, repeating this training at each
  new iteration using the improved predictions. Without any labeled preference
  data, our Self-Taught Evaluator can improve a strong LLM (Llama3-70B-Instruct)
  from 75.4 to 88.3 (88.7 with majority vote) on RewardBench. This outperforms
  commonly used LLM judges such as GPT-4 and matches the performance of the
  top-performing reward models trained with labeled examples.
paper_ai_summary: A self-improving evaluator trained with synthetic data outperforms traditional human-annotated evaluators and matches top-performing reward models on RewardBench.
paper_ai_keywords: LLM-as-a-Judge reasoning traces iterative self-improvement synthetic training data RewardBench Llama3-70B-Instruct GPT-4