title: On scalable oversight with weak LLMs judging strong LLMs
summary: Scalable oversight protocols aim to enable humans to accurately supervise
  superhuman AI. In this paper we study debate, where two AI's compete to
  convince a judge; consultancy, where a single AI tries to convince a judge that
  asks questions; and compare to a baseline of direct question-answering, where
  the judge just answers outright without the AI. We use large language models
  (LLMs) as both AI agents and as stand-ins for human judges, taking the judge
  models to be weaker than agent models. We benchmark on a diverse range of
  asymmetries between judges and agents, extending previous work on a single
  extractive QA task with information asymmetry, to also include mathematics,
  coding, logic and multimodal reasoning asymmetries. We find that debate
  outperforms consultancy across all tasks when the consultant is randomly
  assigned to argue for the correct/incorrect answer. Comparing debate to direct
  question answering, the results depend on the type of task: in extractive QA
  tasks with information asymmetry debate outperforms direct question answering,
  but in other tasks without information asymmetry the results are mixed.
  Previous work assigned debaters/consultants an answer to argue for. When we
  allow them to instead choose which answer to argue for, we find judges are less
  frequently convinced by the wrong answer in debate than in consultancy.
  Further, we find that stronger debater models increase judge accuracy, though
  more modestly than in previous studies.
paper_authors._id: 668b5ea8c2759a7bde1ea83f
paper_authors.name: Zachary Kenton
paper_authors._id: 668b5ea8c2759a7bde1ea840
paper_authors.name: Noah Y. Siegel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-08T21:22:59.957Z
paper_authors._id: 64996a8560738aeea781da8a
paper_authors.avatarUrl: /avatars/902bf170665089fb425a9fe38bcb53d9.svg
paper_authors.fullname: Noah Siegel
paper_authors.type: user
paper_authors.user: noahysiegel
paper_authors._id: 668b5ea8c2759a7bde1ea841
paper_authors.name: János Kramár
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-08T21:22:30.317Z
paper_authors._id: 6647488b45ee22196e798ba6
paper_authors.avatarUrl: /avatars/7c068c30dcd5f07e4d896fd1dce5be2c.svg
paper_authors.fullname: Janos Kramar
paper_authors.type: user
paper_authors.user: jkramar
paper_authors._id: 668b5ea8c2759a7bde1ea842
paper_authors.name: Jonah Brown-Cohen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-08T21:21:44.004Z
paper_authors._id: 667e645f673ab9bcd53c5140
paper_authors.avatarUrl: /avatars/7e32f67f47fc75aa8a33e662375ab464.svg
paper_authors.fullname: Jonah Brown-Cohen
paper_authors.type: user
paper_authors.user: jonahbc
paper_authors._id: 668b5ea8c2759a7bde1ea843
paper_authors.name: Samuel Albanie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-08T21:21:38.140Z
paper_authors._id: 62f3efefd6ba2ee26651f44a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660153837083-noauth.png
paper_authors.fullname: Samuel Albanie
paper_authors.type: user
paper_authors.user: albanie
paper_authors._id: 668b5ea8c2759a7bde1ea844
paper_authors.name: Jannis Bulian
paper_authors._id: 668b5ea8c2759a7bde1ea845
paper_authors.name: Rishabh Agarwal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-08T21:21:11.541Z
paper_authors._id: 65318c1022350ffeaf7726ef
paper_authors.avatarUrl: /avatars/9359e63d38cb69d08da2de07ae933885.svg
paper_authors.fullname: Rishabh Agarwal
paper_authors.type: user
paper_authors.user: rishabhagarwal
paper_authors._id: 668b5ea8c2759a7bde1ea846
paper_authors.name: David Lindner
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-08T21:20:57.375Z
paper_authors._id: 64aef034710608ee1a02b1ac
paper_authors.avatarUrl: /avatars/0f8449bf7ee2a908962502ec863155be.svg
paper_authors.fullname: David Lindner
paper_authors.type: user
paper_authors.user: 1obextiopo
paper_authors._id: 668b5ea8c2759a7bde1ea847
paper_authors.name: Yunhao Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-08T21:20:51.470Z
paper_authors._id: 667c6733ed85453a289a38b2
paper_authors.avatarUrl: /avatars/bf75c134186a8e225500372cbc8c2fcc.svg
paper_authors.fullname: Yunhao Tang
paper_authors.type: user
paper_authors.user: robinphysics
paper_authors._id: 668b5ea8c2759a7bde1ea848
paper_authors.name: Noah D. Goodman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:46:25.192Z
paper_authors._id: 67321274c1f20c742bcf7a8d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ltcQhre6eDRVzn6Vbbyhu.png
paper_authors.fullname: Noah D. Goodman
paper_authors.type: user
paper_authors.user: ngoodman
paper_authors._id: 668b5ea8c2759a7bde1ea849
paper_authors.name: Rohin Shah
paper_title: On scalable oversight with weak LLMs judging strong LLMs
paper_summary: Scalable oversight protocols aim to enable humans to accurately supervise
  superhuman AI. In this paper we study debate, where two AI's compete to
  convince a judge; consultancy, where a single AI tries to convince a judge that
  asks questions; and compare to a baseline of direct question-answering, where
  the judge just answers outright without the AI. We use large language models
  (LLMs) as both AI agents and as stand-ins for human judges, taking the judge
  models to be weaker than agent models. We benchmark on a diverse range of
  asymmetries between judges and agents, extending previous work on a single
  extractive QA task with information asymmetry, to also include mathematics,
  coding, logic and multimodal reasoning asymmetries. We find that debate
  outperforms consultancy across all tasks when the consultant is randomly
  assigned to argue for the correct/incorrect answer. Comparing debate to direct
  question answering, the results depend on the type of task: in extractive QA
  tasks with information asymmetry debate outperforms direct question answering,
  but in other tasks without information asymmetry the results are mixed.
  Previous work assigned debaters/consultants an answer to argue for. When we
  allow them to instead choose which answer to argue for, we find judges are less
  frequently convinced by the wrong answer in debate than in consultancy.
  Further, we find that stronger debater models increase judge accuracy, though
  more modestly than in previous studies.
paper_ai_keywords: large language models (LLMs) debates consultancy direct question-answering information asymmetry multimodal reasoning extractive QA judges