title: SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines
summary: Large language models (LLMs) have demonstrated remarkable proficiency in
  mainstream academic disciplines such as mathematics, physics, and computer
  science. However, human knowledge encompasses over 200 specialized disciplines,
  far exceeding the scope of existing benchmarks. The capabilities of LLMs in
  many of these specialized fields-particularly in light industry, agriculture,
  and service-oriented disciplines-remain inadequately evaluated. To address this
  gap, we present SuperGPQA, a comprehensive benchmark that evaluates
  graduate-level knowledge and reasoning capabilities across 285 disciplines. Our
  benchmark employs a novel Human-LLM collaborative filtering mechanism to
  eliminate trivial or ambiguous questions through iterative refinement based on
  both LLM responses and expert feedback. Our experimental results reveal
  significant room for improvement in the performance of current state-of-the-art
  LLMs across diverse knowledge domains (e.g., the reasoning-focused model
  DeepSeek-R1 achieved the highest accuracy of 61.82% on SuperGPQA), highlighting
  the considerable gap between current model capabilities and artificial general
  intelligence. Additionally, we present comprehensive insights from our
  management of a large-scale annotation process, involving over 80 expert
  annotators and an interactive Human-LLM collaborative system, offering valuable
  methodological guidance for future research initiatives of comparable scope.
paper_authors._id: 67b7efc26348a1df80a8ae53
paper_authors.name: M-A-P Team
paper_authors._id: 67b7efc26348a1df80a8ae54
paper_authors.name: Xinrun Du
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T14:42:53.525Z
paper_authors._id: 654907a4a1faff97850c4eff
paper_authors.avatarUrl: /avatars/458c90151614bc7f116943b6e67d6b8a.svg
paper_authors.fullname: du
paper_authors.type: user
paper_authors.user: dododododo
paper_authors._id: 67b7efc26348a1df80a8ae55
paper_authors.name: Yifan Yao
paper_authors._id: 67b7efc26348a1df80a8ae56
paper_authors.name: Kaijing Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T10:20:18.121Z
paper_authors._id: 65eb65722fbf6807134a636c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65eb65722fbf6807134a636c/8YgrynSM-WExVlu2mwRBY.jpeg
paper_authors.fullname: Kaijing Ma
paper_authors.type: user
paper_authors.user: mkj69
paper_authors._id: 67b7efc26348a1df80a8ae57
paper_authors.name: Bingli Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T10:20:24.792Z
paper_authors._id: 658d0a228cff48d3a4612689
paper_authors.avatarUrl: /avatars/70e297c6cb12d1bdde6d91c23f590b63.svg
paper_authors.fullname: Bingli Wang
paper_authors.type: user
paper_authors.user: BingliW
paper_authors._id: 67b7efc26348a1df80a8ae58
paper_authors.name: Tianyu Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T09:58:24.002Z
paper_authors._id: 64ab99dcb76bfd863eba64c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
paper_authors.fullname: TY.Zheng
paper_authors.type: user
paper_authors.user: aaabiao
paper_authors._id: 67b7efc26348a1df80a8ae59
paper_authors.name: Kang Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T10:20:45.485Z
paper_authors._id: 6578265ddea7e2122d02f6ba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6578265ddea7e2122d02f6ba/Bh6JjoVF5ceLSjV7Z7nTk.jpeg
paper_authors.fullname: kang zhu
paper_authors.type: user
paper_authors.user: kangz
paper_authors._id: 67b7efc26348a1df80a8ae5a
paper_authors.name: Minghao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T09:58:25.894Z
paper_authors._id: 6417d9ea8f689506e7148417
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6417d9ea8f689506e7148417/bAYcruWNw4WvmuQcGgcwC.jpeg
paper_authors.fullname: minghao
paper_authors.type: user
paper_authors.user: Liam-Liu
paper_authors._id: 67b7efc26348a1df80a8ae5b
paper_authors.name: Yiming Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T10:21:08.553Z
paper_authors._id: 6555e8d8a0c34cd61a6b9ce3
paper_authors.avatarUrl: /avatars/71dc562cef4bd42f6b762f036357c800.svg
paper_authors.fullname: yimingliang
paper_authors.type: user
paper_authors.user: yimingliang
paper_authors._id: 67b7efc26348a1df80a8ae5c
paper_authors.name: Xiaolong Jin
paper_authors._id: 67b7efc26348a1df80a8ae5d
paper_authors.name: Zhenlin Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T10:21:31.565Z
paper_authors._id: 67375a6ae6b1d15ff5359a54
paper_authors.avatarUrl: /avatars/9d32d9e3bfb43b8d001c6ddeae720ec5.svg
paper_authors.fullname: weizhenlin
paper_authors.type: user
paper_authors.user: vzl123
paper_authors._id: 67b7efc26348a1df80a8ae5e
paper_authors.name: Chujie Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T09:58:34.124Z
paper_authors._id: 610b70452719facd4ea85e28
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/610b70452719facd4ea85e28/S7nMy7D0Rxq0VIVblhYDG.jpeg
paper_authors.fullname: Chujie Zheng
paper_authors.type: user
paper_authors.user: chujiezheng
paper_authors._id: 67b7efc26348a1df80a8ae5f
paper_authors.name: Kaixing Deng
paper_authors._id: 67b7efc26348a1df80a8ae60
paper_authors.name: Shuyue Guo
paper_authors._id: 67b7efc26348a1df80a8ae61
paper_authors.name: Shian Jia
paper_authors._id: 67b7efc26348a1df80a8ae62
paper_authors.name: Sichao Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T10:22:09.140Z
paper_authors._id: 675085408119fa5fac3cd7cf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/0Mrrkzhv0wBggP5kGKtSt.png
paper_authors.fullname: jiangsichao
paper_authors.type: user
paper_authors.user: jsc137
paper_authors._id: 67b7efc26348a1df80a8ae63
paper_authors.name: Yiyan Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T10:22:15.619Z
paper_authors._id: 67a9d186b22571659007a43d
paper_authors.avatarUrl: /avatars/79b06cf0983083b6161374e66a8c51b2.svg
paper_authors.fullname: Yiyan Liao
paper_authors.type: user
paper_authors.user: yiyanliao
paper_authors._id: 67b7efc26348a1df80a8ae64
paper_authors.name: Rui Li
paper_authors._id: 67b7efc26348a1df80a8ae65
paper_authors.name: Qinrui Li
paper_authors._id: 67b7efc26348a1df80a8ae66
paper_authors.name: Sirun Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T10:22:30.478Z
paper_authors._id: 67ab7826ab5ebf181a7f78d7
paper_authors.avatarUrl: /avatars/d6baf414011d6df659da4eb58e9d8958.svg
paper_authors.fullname: Sirun Li
paper_authors.type: user
paper_authors.user: inorganicwriter
paper_authors._id: 67b7efc26348a1df80a8ae67
paper_authors.name: Yizhi Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T14:42:51.449Z
paper_authors._id: 6382252f54421460665ec501
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6382252f54421460665ec501/gW9fev3T5QPcNq4f9hqB1.jpeg
paper_authors.fullname: Yizhi Li
paper_authors.type: user
paper_authors.user: yizhilll
paper_authors._id: 67b7efc26348a1df80a8ae68
paper_authors.name: Yunwen Li
paper_authors._id: 67b7efc26348a1df80a8ae69
paper_authors.name: Dehua Ma
paper_authors._id: 67b7efc26348a1df80a8ae6a
paper_authors.name: Yuansheng Ni
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T09:58:30.371Z
paper_authors._id: 64de37ee5e192985054be575
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64de37ee5e192985054be575/fVV7JQMtp_J3uFqszJJHH.jpeg
paper_authors.fullname: Yuansheng Ni
paper_authors.type: user
paper_authors.user: yuanshengni
paper_authors._id: 67b7efc26348a1df80a8ae6b
paper_authors.name: Haoran Que
paper_authors._id: 67b7efc26348a1df80a8ae6c
paper_authors.name: Qiyao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T09:58:28.639Z
paper_authors._id: 64560618bfdf9c63ce2d658a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64560618bfdf9c63ce2d658a/GVBWU4yNzRsjdyzKT3z3B.jpeg
paper_authors.fullname: Mathsion Wong
paper_authors.type: user
paper_authors.user: QiYao-Wang
paper_authors._id: 67b7efc26348a1df80a8ae6d
paper_authors.name: Zhoufutu Wen
paper_authors._id: 67b7efc26348a1df80a8ae6e
paper_authors.name: Siwei Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-16T21:18:56.110Z
paper_authors._id: 656d97b10bbc114fe64a96c5
paper_authors.avatarUrl: /avatars/fd23bae1d85c5b96c42064a5ddcfad41.svg
paper_authors.fullname: SiweiWu
paper_authors.type: user
paper_authors.user: SiweiWu
paper_authors._id: 67b7efc26348a1df80a8ae6f
paper_authors.name: Tianshun Xing
paper_authors._id: 67b7efc26348a1df80a8ae70
paper_authors.name: Ming Xu
paper_authors._id: 67b7efc26348a1df80a8ae71
paper_authors.name: Zhenzhu Yang
paper_authors._id: 67b7efc26348a1df80a8ae72
paper_authors.name: Zekun Moore Wang
paper_authors._id: 67b7efc26348a1df80a8ae73
paper_authors.name: Junting Zhou
paper_authors._id: 67b7efc26348a1df80a8ae74
paper_authors.name: Yuelin Bai
paper_authors._id: 67b7efc26348a1df80a8ae75
paper_authors.name: Xingyuan Bu
paper_authors._id: 67b7efc26348a1df80a8ae76
paper_authors.name: Chenglin Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T10:23:17.731Z
paper_authors._id: 64f9c21b681224dbe49a2280
paper_authors.avatarUrl: /avatars/df26cc4b4c6105af2c77392db61e3a27.svg
paper_authors.fullname: caichenglin
paper_authors.type: user
paper_authors.user: easy4mego
paper_authors._id: 67b7efc26348a1df80a8ae77
paper_authors.name: Liang Chen
paper_authors._id: 67b7efc26348a1df80a8ae78
paper_authors.name: Yifan Chen
paper_authors._id: 67b7efc26348a1df80a8ae79
paper_authors.name: Chengtuo Cheng
paper_authors._id: 67b7efc26348a1df80a8ae7a
paper_authors.name: Tianhao Cheng
paper_authors._id: 67b7efc26348a1df80a8ae7b
paper_authors.name: Keyi Ding
paper_authors._id: 67b7efc26348a1df80a8ae7c
paper_authors.name: Siming Huang
paper_authors._id: 67b7efc26348a1df80a8ae7d
paper_authors.name: Yun Huang
paper_authors._id: 67b7efc26348a1df80a8ae7e
paper_authors.name: Yaoru Li
paper_authors._id: 67b7efc26348a1df80a8ae7f
paper_authors.name: Yizhe Li
paper_authors._id: 67b7efc26348a1df80a8ae80
paper_authors.name: Zhaoqun Li
paper_authors._id: 67b7efc26348a1df80a8ae81
paper_authors.name: Tianhao Liang
paper_authors._id: 67b7efc26348a1df80a8ae82
paper_authors.name: Chengdong Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T15:15:33.213Z
paper_authors._id: 66751c722b487c2e015a1f60
paper_authors.avatarUrl: /avatars/d66a98b625451ccea1b4dfcdaf623304.svg
paper_authors.fullname: lin
paper_authors.type: user
paper_authors.user: adams6435
paper_authors._id: 67b7efc26348a1df80a8ae83
paper_authors.name: Hongquan Lin
paper_authors._id: 67b7efc26348a1df80a8ae84
paper_authors.name: Yinghao Ma
paper_authors._id: 67b7efc26348a1df80a8ae85
paper_authors.name: Zhongyuan Peng
paper_authors._id: 67b7efc26348a1df80a8ae86
paper_authors.name: Zifan Peng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T09:58:20.429Z
paper_authors._id: 65adda5299c3bd19c74d6a8d
paper_authors.avatarUrl: /avatars/1ce504b64ab60f375b235ebaf81cafd6.svg
paper_authors.fullname: PENG ZIFAN
paper_authors.type: user
paper_authors.user: Ziffer
paper_authors._id: 67b7efc26348a1df80a8ae87
paper_authors.name: Qige Qi
paper_authors._id: 67b7efc26348a1df80a8ae88
paper_authors.name: Shi Qiu
paper_authors._id: 67b7efc26348a1df80a8ae89
paper_authors.name: Xingwei Qu
paper_authors._id: 67b7efc26348a1df80a8ae8a
paper_authors.name: Yizhou Tan
paper_authors._id: 67b7efc26348a1df80a8ae8b
paper_authors.name: Zili Wang
paper_authors._id: 67b7efc26348a1df80a8ae8c
paper_authors.name: Chenqing Wang
paper_authors._id: 67b7efc26348a1df80a8ae8d
paper_authors.name: Hao Wang
paper_authors._id: 67b7efc26348a1df80a8ae8e
paper_authors.name: Yiya Wang
paper_authors._id: 67b7efc26348a1df80a8ae8f
paper_authors.name: Yubo Wang
paper_authors._id: 67b7efc26348a1df80a8ae90
paper_authors.name: Jiajun Xu
paper_authors._id: 67b7efc26348a1df80a8ae91
paper_authors.name: Kexin Yang
paper_authors._id: 67b7efc26348a1df80a8ae92
paper_authors.name: Ruibin Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T08:44:01.351Z
paper_authors._id: 5fd6f670053c8345eddc1b68
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5fd6f670053c8345eddc1b68/cuTsu2krRYHC6zYGD2dpQ.jpeg
paper_authors.fullname: Ruibin Yuan
paper_authors.type: user
paper_authors.user: a43992899
paper_authors._id: 67b7efc26348a1df80a8ae93
paper_authors.name: Yuanhao Yue
paper_authors._id: 67b7efc26348a1df80a8ae94
paper_authors.name: Tianyang Zhan
paper_authors._id: 67b7efc26348a1df80a8ae95
paper_authors.name: Chun Zhang
paper_authors._id: 67b7efc26348a1df80a8ae96
paper_authors.name: Jingyang Zhang
paper_authors._id: 67b7efc26348a1df80a8ae97
paper_authors.name: Xiyue Zhang
paper_authors._id: 67b7efc26348a1df80a8ae98
paper_authors.name: Xingjian Zhang
paper_authors._id: 67b7efc26348a1df80a8ae99
paper_authors.name: Yue Zhang
paper_authors._id: 67b7efc26348a1df80a8ae9a
paper_authors.name: Yongchi Zhao
paper_authors._id: 67b7efc26348a1df80a8ae9b
paper_authors.name: Xiangyu Zheng
paper_authors._id: 67b7efc26348a1df80a8ae9c
paper_authors.name: Chenghua Zhong
paper_authors._id: 67b7efc26348a1df80a8ae9d
paper_authors.name: Yang Gao
paper_authors._id: 67b7efc26348a1df80a8ae9e
paper_authors.name: Zhoujun Li
paper_authors._id: 67b7efc26348a1df80a8ae9f
paper_authors.name: Dayiheng Liu
paper_authors._id: 67b7efc26348a1df80a8aea0
paper_authors.name: Qian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T09:58:32.399Z
paper_authors._id: 612ee6a7b960e78c6d2319d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
paper_authors.fullname: Qian Liu
paper_authors.type: user
paper_authors.user: SivilTaram
paper_authors._id: 67b7efc26348a1df80a8aea1
paper_authors.name: Tianyu Liu
paper_authors._id: 67b7efc26348a1df80a8aea2
paper_authors.name: Shiwen Ni
paper_authors._id: 67b7efc26348a1df80a8aea3
paper_authors.name: Junran Peng
paper_authors._id: 67b7efc26348a1df80a8aea4
paper_authors.name: Yujia Qin
paper_authors._id: 67b7efc26348a1df80a8aea5
paper_authors.name: Wenbo Su
paper_authors._id: 67b7efc26348a1df80a8aea6
paper_authors.name: Guoyin Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T10:23:25.946Z
paper_authors._id: 6490d4ba1afdee3acd1147f6
paper_authors.avatarUrl: /avatars/ae13c7b21fe9ced7541dcd664d1b94ed.svg
paper_authors.fullname: Guoyin Wang
paper_authors.type: user
paper_authors.user: guoyinwang
paper_authors._id: 67b7efc26348a1df80a8aea7
paper_authors.name: Shi Wang
paper_authors._id: 67b7efc26348a1df80a8aea8
paper_authors.name: Jian Yang
paper_authors._id: 67b7efc26348a1df80a8aea9
paper_authors.name: Min Yang
paper_authors._id: 67b7efc26348a1df80a8aeaa
paper_authors.name: Meng Cao
paper_authors._id: 67b7efc26348a1df80a8aeab
paper_authors.name: Xiang Yue
paper_authors._id: 67b7efc26348a1df80a8aeac
paper_authors.name: Zhaoxiang Zhang
paper_authors._id: 67b7efc26348a1df80a8aead
paper_authors.name: Wangchunshu Zhou
paper_authors._id: 67b7efc26348a1df80a8aeae
paper_authors.name: Jiaheng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T09:58:22.185Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 67b7efc26348a1df80a8aeaf
paper_authors.name: Qunshu Lin
paper_authors._id: 67b7efc26348a1df80a8aeb0
paper_authors.name: Wenhao Huang
paper_authors._id: 67b7efc26348a1df80a8aeb1
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-27T09:17:53.865Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_title: SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines
paper_summary: Large language models (LLMs) have demonstrated remarkable proficiency in
  mainstream academic disciplines such as mathematics, physics, and computer
  science. However, human knowledge encompasses over 200 specialized disciplines,
  far exceeding the scope of existing benchmarks. The capabilities of LLMs in
  many of these specialized fields-particularly in light industry, agriculture,
  and service-oriented disciplines-remain inadequately evaluated. To address this
  gap, we present SuperGPQA, a comprehensive benchmark that evaluates
  graduate-level knowledge and reasoning capabilities across 285 disciplines. Our
  benchmark employs a novel Human-LLM collaborative filtering mechanism to
  eliminate trivial or ambiguous questions through iterative refinement based on
  both LLM responses and expert feedback. Our experimental results reveal
  significant room for improvement in the performance of current state-of-the-art
  LLMs across diverse knowledge domains (e.g., the reasoning-focused model
  DeepSeek-R1 achieved the highest accuracy of 61.82% on SuperGPQA), highlighting
  the considerable gap between current model capabilities and artificial general
  intelligence. Additionally, we present comprehensive insights from our
  management of a large-scale annotation process, involving over 80 expert
  annotators and an interactive Human-LLM collaborative system, offering valuable
  methodological guidance for future research initiatives of comparable scope.