SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines
Large language models (LLMs) have demonstrated remarkable proficiency in
mainstream academic disciplines such as mathematics, physics, and computer
science. However, human knowledge encompasses over 200 specialized disciplines,
far exceeding the scope of existing benchmarks. The capabilities of LLMs in
many of these specialized fields-particularly in light industry, agriculture,
and service-oriented disciplines-remain inadequately evaluated. To address this
gap, we present SuperGPQA, a comprehensive benchmark that evaluates
graduate-level knowledge and reasoning capabilities across 285 disciplines. Our
benchmark employs a novel Human-LLM collaborative filtering mechanism to
eliminate trivial or ambiguous questions through iterative refinement based on
both LLM responses and expert feedback. Our experimental results reveal
significant room for improvement in the performance of current state-of-the-art
LLMs across diverse knowledge domains (e.g., the reasoning-focused model
DeepSeek-R1 achieved the highest accuracy of 61.82% on SuperGPQA), highlighting
the considerable gap between current model capabilities and artificial general
intelligence. Additionally, we present comprehensive insights from our
management of a large-scale annotation process, involving over 80 expert
annotators and an interactive Human-LLM collaborative system, offering valuable
methodological guidance for future research initiatives of comparable scope.
67b7efc26348a1df80a8ae53
M-A-P Team
67b7efc26348a1df80a8ae54
Xinrun Du
claimed_verified
2025-02-21T14:42:53.525Z
654907a4a1faff97850c4eff
/avatars/458c90151614bc7f116943b6e67d6b8a.svg
du
user
dododododo
67b7efc26348a1df80a8ae55
Yifan Yao
67b7efc26348a1df80a8ae56
Kaijing Ma
admin_assigned
2025-02-21T10:20:18.121Z
65eb65722fbf6807134a636c
https://cdn-avatars.huggingface.co/v1/production/uploads/65eb65722fbf6807134a636c/8YgrynSM-WExVlu2mwRBY.jpeg
Kaijing Ma
user
mkj69
67b7efc26348a1df80a8ae57
Bingli Wang
admin_assigned
2025-02-21T10:20:24.792Z
658d0a228cff48d3a4612689
/avatars/70e297c6cb12d1bdde6d91c23f590b63.svg
Bingli Wang
user
BingliW
67b7efc26348a1df80a8ae58
Tianyu Zheng
claimed_verified
2025-02-21T09:58:24.002Z
64ab99dcb76bfd863eba64c1
https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
TY.Zheng
user
aaabiao
67b7efc26348a1df80a8ae59
Kang Zhu
admin_assigned
2025-02-21T10:20:45.485Z
6578265ddea7e2122d02f6ba
https://cdn-avatars.huggingface.co/v1/production/uploads/6578265ddea7e2122d02f6ba/Bh6JjoVF5ceLSjV7Z7nTk.jpeg
kang zhu
user
kangz
67b7efc26348a1df80a8ae5a
Minghao Liu
claimed_verified
2025-02-21T09:58:25.894Z
6417d9ea8f689506e7148417
https://cdn-avatars.huggingface.co/v1/production/uploads/6417d9ea8f689506e7148417/bAYcruWNw4WvmuQcGgcwC.jpeg
minghao
user
Liam-Liu
67b7efc26348a1df80a8ae5b
Yiming Liang
admin_assigned
2025-02-21T10:21:08.553Z
6555e8d8a0c34cd61a6b9ce3
/avatars/71dc562cef4bd42f6b762f036357c800.svg
yimingliang
user
yimingliang
67b7efc26348a1df80a8ae5c
Xiaolong Jin
67b7efc26348a1df80a8ae5d
Zhenlin Wei
admin_assigned
2025-02-21T10:21:31.565Z
67375a6ae6b1d15ff5359a54
/avatars/9d32d9e3bfb43b8d001c6ddeae720ec5.svg
weizhenlin
user
vzl123
67b7efc26348a1df80a8ae5e
Chujie Zheng
claimed_verified
2025-02-21T09:58:34.124Z
610b70452719facd4ea85e28
https://cdn-avatars.huggingface.co/v1/production/uploads/610b70452719facd4ea85e28/S7nMy7D0Rxq0VIVblhYDG.jpeg
Chujie Zheng
user
chujiezheng
67b7efc26348a1df80a8ae5f
Kaixing Deng
67b7efc26348a1df80a8ae60
Shuyue Guo
67b7efc26348a1df80a8ae61
Shian Jia
67b7efc26348a1df80a8ae62
Sichao Jiang
admin_assigned
2025-02-21T10:22:09.140Z
675085408119fa5fac3cd7cf
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/0Mrrkzhv0wBggP5kGKtSt.png
jiangsichao
user
jsc137
67b7efc26348a1df80a8ae63
Yiyan Liao
admin_assigned
2025-02-21T10:22:15.619Z
67a9d186b22571659007a43d
/avatars/79b06cf0983083b6161374e66a8c51b2.svg
Yiyan Liao
user
yiyanliao
67b7efc26348a1df80a8ae64
Rui Li
67b7efc26348a1df80a8ae65
Qinrui Li
67b7efc26348a1df80a8ae66
Sirun Li
admin_assigned
2025-02-21T10:22:30.478Z
67ab7826ab5ebf181a7f78d7
/avatars/d6baf414011d6df659da4eb58e9d8958.svg
Sirun Li
user
inorganicwriter
67b7efc26348a1df80a8ae67
Yizhi Li
claimed_verified
2025-02-21T14:42:51.449Z
6382252f54421460665ec501
https://cdn-avatars.huggingface.co/v1/production/uploads/6382252f54421460665ec501/gW9fev3T5QPcNq4f9hqB1.jpeg
Yizhi Li
user
yizhilll
67b7efc26348a1df80a8ae68
Yunwen Li
67b7efc26348a1df80a8ae69
Dehua Ma
67b7efc26348a1df80a8ae6a
Yuansheng Ni
claimed_verified
2025-02-21T09:58:30.371Z
64de37ee5e192985054be575
https://cdn-avatars.huggingface.co/v1/production/uploads/64de37ee5e192985054be575/fVV7JQMtp_J3uFqszJJHH.jpeg
Yuansheng Ni
user
yuanshengni
67b7efc26348a1df80a8ae6b
Haoran Que
67b7efc26348a1df80a8ae6c
Qiyao Wang
claimed_verified
2025-02-21T09:58:28.639Z
64560618bfdf9c63ce2d658a
https://cdn-avatars.huggingface.co/v1/production/uploads/64560618bfdf9c63ce2d658a/GVBWU4yNzRsjdyzKT3z3B.jpeg
Mathsion Wong
user
QiYao-Wang
67b7efc26348a1df80a8ae6d
Zhoufutu Wen
67b7efc26348a1df80a8ae6e
Siwei Wu
claimed_verified
2025-03-16T21:18:56.110Z
656d97b10bbc114fe64a96c5
/avatars/fd23bae1d85c5b96c42064a5ddcfad41.svg
SiweiWu
user
SiweiWu
67b7efc26348a1df80a8ae6f
Tianshun Xing
claimed_verified
2025-07-10T09:13:09.320Z
65d2251f98b4a470bf6a26e3
https://cdn-avatars.huggingface.co/v1/production/uploads/65d2251f98b4a470bf6a26e3/C4T0LHYGejrI9mu_k3M8p.jpeg
xts
user
xtsssss
67b7efc26348a1df80a8ae70
Ming Xu
67b7efc26348a1df80a8ae71
Zhenzhu Yang
67b7efc26348a1df80a8ae72
Zekun Moore Wang
67b7efc26348a1df80a8ae73
Junting Zhou
67b7efc26348a1df80a8ae74
Yuelin Bai
67b7efc26348a1df80a8ae75
Xingyuan Bu
claimed_verified
2025-04-09T14:37:08.706Z
6444e7765691ca69b0d95856
/avatars/4ae5001e7c7dea89c4deaf2b05436857.svg
Xingyuan Bu
user
sefira32
67b7efc26348a1df80a8ae76
Chenglin Cai
admin_assigned
2025-02-21T10:23:17.731Z
64f9c21b681224dbe49a2280
/avatars/df26cc4b4c6105af2c77392db61e3a27.svg
caichenglin
user
easy4mego
67b7efc26348a1df80a8ae77
Liang Chen
67b7efc26348a1df80a8ae78
Yifan Chen
67b7efc26348a1df80a8ae79
Chengtuo Cheng
67b7efc26348a1df80a8ae7a
Tianhao Cheng
67b7efc26348a1df80a8ae7b
Keyi Ding
67b7efc26348a1df80a8ae7c
Siming Huang
67b7efc26348a1df80a8ae7d
Yun Huang
67b7efc26348a1df80a8ae7e
Yaoru Li
67b7efc26348a1df80a8ae7f
Yizhe Li
67b7efc26348a1df80a8ae80
Zhaoqun Li
67b7efc26348a1df80a8ae81
Tianhao Liang
claimed_verified
2025-04-11T08:23:45.498Z
6703ec213df5fe425086ef73
/avatars/e6f9dad6587ee0883ae10f8805ab7ea9.svg
Tianhao Liang
user
tianhao2k
67b7efc26348a1df80a8ae82
Chengdong Lin
claimed_verified
2025-02-21T15:15:33.213Z
66751c722b487c2e015a1f60
/avatars/d66a98b625451ccea1b4dfcdaf623304.svg
lin
user
adams6435
67b7efc26348a1df80a8ae83
Hongquan Lin
67b7efc26348a1df80a8ae84
Yinghao Ma
67b7efc26348a1df80a8ae85
Zhongyuan Peng
claimed_verified
2025-07-09T15:11:42.455Z
63299f93688ad82b783aaf20
/avatars/7c11e60e551ef1c62aa2862529e357f5.svg
zhongyuan peng
user
happzy2633
67b7efc26348a1df80a8ae86
Zifan Peng
claimed_verified
2025-02-21T09:58:20.429Z
65adda5299c3bd19c74d6a8d
/avatars/1ce504b64ab60f375b235ebaf81cafd6.svg
PENG ZIFAN
user
Ziffer
67b7efc26348a1df80a8ae87
Qige Qi
67b7efc26348a1df80a8ae88
Shi Qiu
claimed_verified
2025-04-25T08:36:12.459Z
6644bb2c9bdbd85493074411
https://cdn-avatars.huggingface.co/v1/production/uploads/6644bb2c9bdbd85493074411/iug8dkP1zjID-kXzNxDjD.jpeg
SHI QIU
user
StarThomas1002
67b7efc26348a1df80a8ae89
Xingwei Qu
67b7efc26348a1df80a8ae8a
Yizhou Tan
67b7efc26348a1df80a8ae8b
Zili Wang
67b7efc26348a1df80a8ae8c
Chenqing Wang
67b7efc26348a1df80a8ae8d
Hao Wang
67b7efc26348a1df80a8ae8e
Yiya Wang
67b7efc26348a1df80a8ae8f
Yubo Wang
claimed_verified
2025-04-08T06:58:21.107Z
636a35eff8d9af4aea181608
https://cdn-avatars.huggingface.co/v1/production/uploads/636a35eff8d9af4aea181608/s9GFJYd_QXVbg0Lb4JpKj.jpeg
yubo
user
ubowang
67b7efc26348a1df80a8ae90
Jiajun Xu
67b7efc26348a1df80a8ae91
Kexin Yang
67b7efc26348a1df80a8ae92
Ruibin Yuan
claimed_verified
2025-03-12T08:44:01.351Z
5fd6f670053c8345eddc1b68
https://cdn-avatars.huggingface.co/v1/production/uploads/5fd6f670053c8345eddc1b68/cuTsu2krRYHC6zYGD2dpQ.jpeg
Ruibin Yuan
user
a43992899
67b7efc26348a1df80a8ae93
Yuanhao Yue
67b7efc26348a1df80a8ae94
Tianyang Zhan
67b7efc26348a1df80a8ae95
Chun Zhang
67b7efc26348a1df80a8ae96
Jingyang Zhang
67b7efc26348a1df80a8ae97
Xiyue Zhang
67b7efc26348a1df80a8ae98
Xingjian Zhang
67b7efc26348a1df80a8ae99
Yue Zhang
67b7efc26348a1df80a8ae9a
Yongchi Zhao
67b7efc26348a1df80a8ae9b
Xiangyu Zheng
67b7efc26348a1df80a8ae9c
Chenghua Zhong
67b7efc26348a1df80a8ae9d
Yang Gao
67b7efc26348a1df80a8ae9e
Zhoujun Li
67b7efc26348a1df80a8ae9f
Dayiheng Liu
67b7efc26348a1df80a8aea0
Qian Liu
claimed_verified
2025-02-21T09:58:32.399Z
612ee6a7b960e78c6d2319d4
https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
Qian Liu
user
SivilTaram
67b7efc26348a1df80a8aea1
Tianyu Liu
67b7efc26348a1df80a8aea2
Shiwen Ni
67b7efc26348a1df80a8aea3
Junran Peng
67b7efc26348a1df80a8aea4
Yujia Qin
67b7efc26348a1df80a8aea5
Wenbo Su
67b7efc26348a1df80a8aea6
Guoyin Wang
admin_assigned
2025-02-21T10:23:25.946Z
6490d4ba1afdee3acd1147f6
/avatars/ae13c7b21fe9ced7541dcd664d1b94ed.svg
Guoyin Wang
user
guoyinwang
67b7efc26348a1df80a8aea7
Shi Wang
67b7efc26348a1df80a8aea8
Jian Yang
67b7efc26348a1df80a8aea9
Min Yang
67b7efc26348a1df80a8aeaa
Meng Cao
67b7efc26348a1df80a8aeab
Xiang Yue
67b7efc26348a1df80a8aeac
Zhaoxiang Zhang
67b7efc26348a1df80a8aead
Wangchunshu Zhou
claimed_verified
2025-06-18T12:17:59.715Z
628c8598ef14f971b698107f
/avatars/3a4ad87e6b5f9e836a1160d869df1447.svg
Zhou
user
Wangchunshu
67b7efc26348a1df80a8aeae
Jiaheng Liu
claimed_verified
2025-02-21T09:58:22.185Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
67b7efc26348a1df80a8aeaf
Qunshu Lin
67b7efc26348a1df80a8aeb0
Wenhao Huang
67b7efc26348a1df80a8aeb1
Ge Zhang
claimed_verified
2025-02-27T09:17:53.865Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines
Large language models (LLMs) have demonstrated remarkable proficiency in
mainstream academic disciplines such as mathematics, physics, and computer
science. However, human knowledge encompasses over 200 specialized disciplines,
far exceeding the scope of existing benchmarks. The capabilities of LLMs in
many of these specialized fields-particularly in light industry, agriculture,
and service-oriented disciplines-remain inadequately evaluated. To address this
gap, we present SuperGPQA, a comprehensive benchmark that evaluates
graduate-level knowledge and reasoning capabilities across 285 disciplines. Our
benchmark employs a novel Human-LLM collaborative filtering mechanism to
eliminate trivial or ambiguous questions through iterative refinement based on
both LLM responses and expert feedback. Our experimental results reveal
significant room for improvement in the performance of current state-of-the-art
LLMs across diverse knowledge domains (e.g., the reasoning-focused model
DeepSeek-R1 achieved the highest accuracy of 61.82% on SuperGPQA), highlighting
the considerable gap between current model capabilities and artificial general
intelligence. Additionally, we present comprehensive insights from our
management of a large-scale annotation process, involving over 80 expert
annotators and an interactive Human-LLM collaborative system, offering valuable
methodological guidance for future research initiatives of comparable scope.
SuperGPQA, a benchmark evaluating LLMs across 285 disciplines, reveals performance gaps and offers insights into the collaborative filtering process with human experts.
LLMs Large language models SuperGPQA Human-LLM collaborative filtering DeepSeek-R1