title: GPQA: A Graduate-Level Google-Proof Q&A Benchmark
summary: We present GPQA, a challenging dataset of 448 multiple-choice questions
  written by domain experts in biology, physics, and chemistry. We ensure that
  the questions are high-quality and extremely difficult: experts who have or are
  pursuing PhDs in the corresponding domains reach 65% accuracy (74% when
  discounting clear mistakes the experts identified in retrospect), while highly
  skilled non-expert validators only reach 34% accuracy, despite spending on
  average over 30 minutes with unrestricted access to the web (i.e., the
  questions are "Google-proof"). The questions are also difficult for
  state-of-the-art AI systems, with our strongest GPT-4 based baseline achieving
  39% accuracy. If we are to use future AI systems to help us answer very hard
  questions, for example, when developing new scientific knowledge, we need to
  develop scalable oversight methods that enable humans to supervise their
  outputs, which may be difficult even if the supervisors are themselves skilled
  and knowledgeable. The difficulty of GPQA both for skilled non-experts and
  frontier AI systems should enable realistic scalable oversight experiments,
  which we hope can help devise ways for human experts to reliably get truthful
  information from AI systems that surpass human capabilities.
paper_authors._id: 655c17afe8a8971e89a60e2c
paper_authors.name: David Rein
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-24T02:49:26.645Z
paper_authors._id: 655cc37f9af37bae32110d53
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/655cc37f9af37bae32110d53/cFGjZk0Y8gwDrwmRF83oO.jpeg
paper_authors.fullname: David Rein
paper_authors.type: user
paper_authors.user: Idavidrein
paper_authors._id: 655c17afe8a8971e89a60e2d
paper_authors.name: Betty Li Hou
paper_authors._id: 655c17afe8a8971e89a60e2e
paper_authors.name: Asa Cooper Stickland
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-21T13:58:38.252Z
paper_authors._id: 6079eec3c5e1e416b294ea4f
paper_authors.avatarUrl: /avatars/9672fb4c86ec5b20155e2cd941a4edf8.svg
paper_authors.fullname: Asa Cooper Stickland
paper_authors.type: user
paper_authors.user: Asa
paper_authors._id: 655c17afe8a8971e89a60e2f
paper_authors.name: Jackson Petty
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-21T14:44:27.130Z
paper_authors._id: 655cbcde95959f36af636d9a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/655cbcde95959f36af636d9a/wV-pwUpUeDOjX9iUNa9Zq.jpeg
paper_authors.fullname: Jackson Petty
paper_authors.type: user
paper_authors.user: jowenpetty
paper_authors._id: 655c17afe8a8971e89a60e30
paper_authors.name: Richard Yuanzhe Pang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-21T14:44:24.216Z
paper_authors._id: 64d50fa59a6a7ae984d90852
paper_authors.avatarUrl: /avatars/26b78c151e707e3f9a9516c877d18ff7.svg
paper_authors.fullname: Richard Pang
paper_authors.type: user
paper_authors.user: yzpang
paper_authors._id: 655c17afe8a8971e89a60e31
paper_authors.name: Julien Dirani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-21T13:58:02.371Z
paper_authors._id: 638a7bd8fb4c3732d06926ac
paper_authors.avatarUrl: /avatars/33c5d04d524033249853670b7a74ac99.svg
paper_authors.fullname: Julien Dirani
paper_authors.type: user
paper_authors.user: jdirani
paper_authors._id: 655c17afe8a8971e89a60e32
paper_authors.name: Julian Michael
paper_authors._id: 655c17afe8a8971e89a60e33
paper_authors.name: Samuel R. Bowman
paper_title: GPQA: A Graduate-Level Google-Proof Q&A Benchmark
paper_summary: We present GPQA, a challenging dataset of 448 multiple-choice questions
  written by domain experts in biology, physics, and chemistry. We ensure that
  the questions are high-quality and extremely difficult: experts who have or are
  pursuing PhDs in the corresponding domains reach 65% accuracy (74% when
  discounting clear mistakes the experts identified in retrospect), while highly
  skilled non-expert validators only reach 34% accuracy, despite spending on
  average over 30 minutes with unrestricted access to the web (i.e., the
  questions are "Google-proof"). The questions are also difficult for
  state-of-the-art AI systems, with our strongest GPT-4 based baseline achieving
  39% accuracy. If we are to use future AI systems to help us answer very hard
  questions, for example, when developing new scientific knowledge, we need to
  develop scalable oversight methods that enable humans to supervise their
  outputs, which may be difficult even if the supervisors are themselves skilled
  and knowledgeable. The difficulty of GPQA both for skilled non-experts and
  frontier AI systems should enable realistic scalable oversight experiments,
  which we hope can help devise ways for human experts to reliably get truthful
  information from AI systems that surpass human capabilities.
paper_ai_summary: A dataset of extremely difficult multiple-choice questions challenges both experts and AI systems, facilitating the development of scalable oversight methods for AI-generated knowledge.
paper_ai_keywords: GPQA multiple-choice questions domain experts biology physics chemistry GPT-4 AI systems scalable oversight human supervision truthful information