title: Great Models Think Alike and this Undermines AI Oversight
summary: As Language Model (LM) capabilities advance, evaluating and supervising them
  at scale is getting harder for humans. There is hope that other language models
  can automate both these tasks, which we refer to as "AI Oversight". We study
  how model similarity affects both aspects of AI oversight by proposing a
  probabilistic metric for LM similarity based on overlap in model mistakes.
  Using this metric, we first show that LLM-as-a-judge scores favor models
  similar to the judge, generalizing recent self-preference results. Then, we
  study training on LM annotations, and find complementary knowledge between the
  weak supervisor and strong student model plays a crucial role in gains from
  "weak-to-strong generalization". As model capabilities increase, it becomes
  harder to find their mistakes, and we might defer more to AI oversight.
  However, we observe a concerning trend -- model mistakes are becoming more
  similar with increasing capabilities, pointing to risks from correlated
  failures. Our work underscores the importance of reporting and correcting for
  model similarity, especially in the emerging paradigm of AI oversight.
paper_authors._id: 67a5b9107897c8f5406155e0
paper_authors.name: Shashwat Goel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:39:36.508Z
paper_authors._id: 6506832221ac448013f94995
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6506832221ac448013f94995/sVUI1JV4Dxan5l-MqNze4.jpeg
paper_authors.fullname: Shashwat Goel
paper_authors.type: user
paper_authors.user: shash42
paper_authors._id: 67a5b9107897c8f5406155e1
paper_authors.name: Joschka Struber
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:39:43.250Z
paper_authors._id: 6728c6113d35dd53cfe9f30c
paper_authors.avatarUrl: /avatars/7f93b9d41446cce382f63c78ca5059a1.svg
paper_authors.fullname: Joschka Strüber
paper_authors.type: user
paper_authors.user: Klingspor
paper_authors._id: 67a5b9107897c8f5406155e2
paper_authors.name: Ilze Amanda Auzina
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:40:04.242Z
paper_authors._id: 671b49503fd1d03dc69194b0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/tnkR0j1VaWClUcumXcgjQ.png
paper_authors.fullname: Ilze Amanda Auzina
paper_authors.type: user
paper_authors.user: iaa01
paper_authors._id: 67a5b9107897c8f5406155e3
paper_authors.name: Karuna K Chandra
paper_authors._id: 67a5b9107897c8f5406155e4
paper_authors.name: Ponnurangam Kumaraguru
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-08T13:56:22.133Z
paper_authors._id: 67a6a4b7f379cef464950268
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/yaMsT2jYsj6mvvYU1gvO_.jpeg
paper_authors.fullname: ponnurangam kumaraguru
paper_authors.type: user
paper_authors.user: pk-profgiri
paper_authors._id: 67a5b9107897c8f5406155e5
paper_authors.name: Douwe Kiela
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:40:22.153Z
paper_authors._id: 61dc997715b47073db1620dc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1641847245435-61dc997715b47073db1620dc.jpeg
paper_authors.fullname: Douwe Kiela
paper_authors.type: user
paper_authors.user: douwekiela
paper_authors._id: 67a5b9107897c8f5406155e6
paper_authors.name: Ameya Prabhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:40:34.763Z
paper_authors._id: 6464a0d41683d3c81f51924a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6464a0d41683d3c81f51924a/s7yYVwfUB4WOhVFJS6A6T.jpeg
paper_authors.fullname: Ameya Prabhu
paper_authors.type: user
paper_authors.user: AmeyaPrabhu
paper_authors._id: 67a5b9107897c8f5406155e7
paper_authors.name: Matthias Bethge
paper_authors._id: 67a5b9107897c8f5406155e8
paper_authors.name: Jonas Geiping
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:40:49.233Z
paper_authors._id: 63d86dbf3130cadcaf8bdd11
paper_authors.avatarUrl: /avatars/29d79a0c6dcec01111ef192fecd0fa7a.svg
paper_authors.fullname: Jonas Geiping
paper_authors.type: user
paper_authors.user: JonasGeiping
paper_title: Great Models Think Alike and this Undermines AI Oversight
paper_summary: As Language Model (LM) capabilities advance, evaluating and supervising them
  at scale is getting harder for humans. There is hope that other language models
  can automate both these tasks, which we refer to as "AI Oversight". We study
  how model similarity affects both aspects of AI oversight by proposing a
  probabilistic metric for LM similarity based on overlap in model mistakes.
  Using this metric, we first show that LLM-as-a-judge scores favor models
  similar to the judge, generalizing recent self-preference results. Then, we
  study training on LM annotations, and find complementary knowledge between the
  weak supervisor and strong student model plays a crucial role in gains from
  "weak-to-strong generalization". As model capabilities increase, it becomes
  harder to find their mistakes, and we might defer more to AI oversight.
  However, we observe a concerning trend -- model mistakes are becoming more
  similar with increasing capabilities, pointing to risks from correlated
  failures. Our work underscores the importance of reporting and correcting for
  model similarity, especially in the emerging paradigm of AI oversight.
paper_ai_keywords: AI Oversight probabilistic metric LM similarity model mistakes LLM-as-a-judge self-preference weak supervision strong student model weak-to-strong generalization correlated failures