MMLU-Pro: A More Robust and Challenging Multi-Task Language
  Understanding Benchmark
In the age of large-scale language models, benchmarks like the Massive
Multitask Language Understanding (MMLU) have been pivotal in pushing the
boundaries of what AI can achieve in language comprehension and reasoning
across diverse domains. However, as models continue to improve, their
performance on these benchmarks has begun to plateau, making it increasingly
difficult to discern differences in model capabilities. This paper introduces
MMLU-Pro, an enhanced dataset designed to extend the mostly knowledge-driven
MMLU benchmark by integrating more challenging, reasoning-focused questions and
expanding the choice set from four to ten options. Additionally, MMLU-Pro
eliminates the trivial and noisy questions in MMLU. Our experimental results
show that MMLU-Pro not only raises the challenge, causing a significant drop in
accuracy by 16% to 33% compared to MMLU but also demonstrates greater stability
under varying prompts. With 24 different prompt styles tested, the sensitivity
of model scores to prompt variations decreased from 4-5% in MMLU to just 2% in
MMLU-Pro. Additionally, we found that models utilizing Chain of Thought (CoT)
reasoning achieved better performance on MMLU-Pro compared to direct answering,
which is in stark contrast to the findings on the original MMLU, indicating
that MMLU-Pro includes more complex reasoning questions. Our assessments
confirm that MMLU-Pro is a more discriminative benchmark to better track
progress in the field.
665e92a30eb022e5364b0fae
Yubo Wang
claimed_verified
2024-06-14T07:24:48.828Z
636a35eff8d9af4aea181608
https://cdn-avatars.huggingface.co/v1/production/uploads/636a35eff8d9af4aea181608/s9GFJYd_QXVbg0Lb4JpKj.jpeg
yubo
user
ubowang
665e92a30eb022e5364b0faf
Xueguang Ma
admin_assigned
2024-06-04T07:32:53.777Z
5ec82854968f6028e0559f70
/avatars/45b58d912f7d00cb351947cd79d5eeb4.svg
Xueguang Ma
user
MrLight
665e92a30eb022e5364b0fb0
Ge Zhang
admin_assigned
2024-06-04T07:33:45.882Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
665e92a30eb022e5364b0fb1
Yuansheng Ni
admin_assigned
2024-06-04T07:33:55.265Z
64de37ee5e192985054be575
https://cdn-avatars.huggingface.co/v1/production/uploads/64de37ee5e192985054be575/fVV7JQMtp_J3uFqszJJHH.jpeg
Yuansheng Ni
user
yuanshengni
665e92a30eb022e5364b0fb2
Abhranil Chandra
admin_assigned
2024-06-04T07:34:21.474Z
6154ee656181394cc00cb990
/avatars/25d3b6911c5991b1869f5d76ca2c4069.svg
Abhranil Chandra
user
abhranil14
665e92a30eb022e5364b0fb3
Shiguang Guo
665e92a30eb022e5364b0fb4
Weiming Ren
admin_assigned
2024-06-04T07:34:49.769Z
64405a9d518271b0d1beea38
/avatars/b702474588fd7090773320422417a582.svg
Weiming Ren
user
wren93
665e92a30eb022e5364b0fb5
Aaran Arulraj
admin_assigned
2024-06-04T07:35:11.005Z
65fb7212fc9132a2dfb80061
/avatars/7955a4c96cba4e5b25481943f7b84be1.svg
Aaran Arulraj
user
AaranArulraj
665e92a30eb022e5364b0fb6
Xuan He
admin_assigned
2024-06-04T07:35:50.254Z
655c3953ce055ed40a7de0ba
/avatars/70aad195f6aff83279ef1b1f8859419d.svg
Xuan He
user
hexuan21
665e92a30eb022e5364b0fb7
Ziyan Jiang
claimed_verified
2024-06-24T07:33:35.456Z
64778fb8168cb428e00f69b0
https://cdn-avatars.huggingface.co/v1/production/uploads/64778fb8168cb428e00f69b0/D_XYg74zHG9K3HUJj_gD4.jpeg
Ziyan Jiang
user
ziyjiang
665e92a30eb022e5364b0fb8
Tianle Li
admin_assigned
2024-06-04T07:36:51.642Z
63daf95a9f2687298a110386
/avatars/7e922ca76689de8e0e0c4af350115246.svg
Tianle LI
user
tianleliphoebe
665e92a30eb022e5364b0fb9
Max Ku
claimed_verified
2024-06-04T07:31:33.369Z
631d760344503b7227837242
https://cdn-avatars.huggingface.co/v1/production/uploads/631d760344503b7227837242/3b6JRusFX6GKJpsN9ZdeJ.png
Max Ku
user
vinesmsuic
665e92a30eb022e5364b0fba
Kai Wang
665e92a30eb022e5364b0fbb
Alex Zhuang
admin_assigned
2024-06-04T07:52:15.887Z
6498eecc3756a91498f08f60
/avatars/d92a98535d212c73b9249aa48c717f5e.svg
Zhuang
user
AlexZhuang
665e92a30eb022e5364b0fbc
Rongqi Fan
665e92a30eb022e5364b0fbd
Xiang Yue
admin_assigned
2024-06-04T07:52:37.143Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
665e92a30eb022e5364b0fbe
Wenhu Chen
admin_assigned
2024-06-04T07:52:43.625Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
MMLU-Pro: A More Robust and Challenging Multi-Task Language
  Understanding Benchmark
In the age of large-scale language models, benchmarks like the Massive
Multitask Language Understanding (MMLU) have been pivotal in pushing the
boundaries of what AI can achieve in language comprehension and reasoning
across diverse domains. However, as models continue to improve, their
performance on these benchmarks has begun to plateau, making it increasingly
difficult to discern differences in model capabilities. This paper introduces
MMLU-Pro, an enhanced dataset designed to extend the mostly knowledge-driven
MMLU benchmark by integrating more challenging, reasoning-focused questions and
expanding the choice set from four to ten options. Additionally, MMLU-Pro
eliminates the trivial and noisy questions in MMLU. Our experimental results
show that MMLU-Pro not only raises the challenge, causing a significant drop in
accuracy by 16% to 33% compared to MMLU but also demonstrates greater stability
under varying prompts. With 24 different prompt styles tested, the sensitivity
of model scores to prompt variations decreased from 4-5% in MMLU to just 2% in
MMLU-Pro. Additionally, we found that models utilizing Chain of Thought (CoT)
reasoning achieved better performance on MMLU-Pro compared to direct answering,
which is in stark contrast to the findings on the original MMLU, indicating
that MMLU-Pro includes more complex reasoning questions. Our assessments
confirm that MMLU-Pro is a more discriminative benchmark to better track
progress in the field.
MMLU-Pro extends the MMLU benchmark with more challenging reasoning questions, eliminates trivial ones, and demonstrates better stability and discriminative power for language models.
Chain of Thought reasoning-focused questions