title: MMLU-Pro: A More Robust and Challenging Multi-Task Language
    Understanding Benchmark
summary: In the age of large-scale language models, benchmarks like the Massive
  Multitask Language Understanding (MMLU) have been pivotal in pushing the
  boundaries of what AI can achieve in language comprehension and reasoning
  across diverse domains. However, as models continue to improve, their
  performance on these benchmarks has begun to plateau, making it increasingly
  difficult to discern differences in model capabilities. This paper introduces
  MMLU-Pro, an enhanced dataset designed to extend the mostly knowledge-driven
  MMLU benchmark by integrating more challenging, reasoning-focused questions and
  expanding the choice set from four to ten options. Additionally, MMLU-Pro
  eliminates the trivial and noisy questions in MMLU. Our experimental results
  show that MMLU-Pro not only raises the challenge, causing a significant drop in
  accuracy by 16% to 33% compared to MMLU but also demonstrates greater stability
  under varying prompts. With 24 different prompt styles tested, the sensitivity
  of model scores to prompt variations decreased from 4-5% in MMLU to just 2% in
  MMLU-Pro. Additionally, we found that models utilizing Chain of Thought (CoT)
  reasoning achieved better performance on MMLU-Pro compared to direct answering,
  which is in stark contrast to the findings on the original MMLU, indicating
  that MMLU-Pro includes more complex reasoning questions. Our assessments
  confirm that MMLU-Pro is a more discriminative benchmark to better track
  progress in the field.
paper_authors._id: 665e92a30eb022e5364b0fae
paper_authors.name: Yubo Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T07:24:48.828Z
paper_authors._id: 636a35eff8d9af4aea181608
paper_authors.avatarUrl: /avatars/d9c5cf3491243d1f2b1c5df1873ee8e7.svg
paper_authors.fullname: yubo
paper_authors.type: user
paper_authors.user: ubowang
paper_authors._id: 665e92a30eb022e5364b0faf
paper_authors.name: Xueguang Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T07:32:53.777Z
paper_authors._id: 5ec82854968f6028e0559f70
paper_authors.avatarUrl: /avatars/45b58d912f7d00cb351947cd79d5eeb4.svg
paper_authors.fullname: Xueguang Ma
paper_authors.type: user
paper_authors.user: MrLight
paper_authors._id: 665e92a30eb022e5364b0fb0
paper_authors.name: Ge Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T07:33:45.882Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 665e92a30eb022e5364b0fb1
paper_authors.name: Yuansheng Ni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T07:33:55.265Z
paper_authors._id: 64de37ee5e192985054be575
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64de37ee5e192985054be575/fVV7JQMtp_J3uFqszJJHH.jpeg
paper_authors.fullname: Yuansheng Ni
paper_authors.type: user
paper_authors.user: yuanshengni
paper_authors._id: 665e92a30eb022e5364b0fb2
paper_authors.name: Abhranil Chandra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T07:34:21.474Z
paper_authors._id: 6154ee656181394cc00cb990
paper_authors.avatarUrl: /avatars/25d3b6911c5991b1869f5d76ca2c4069.svg
paper_authors.fullname: Abhranil Chandra
paper_authors.type: user
paper_authors.user: abhranil14
paper_authors._id: 665e92a30eb022e5364b0fb3
paper_authors.name: Shiguang Guo
paper_authors._id: 665e92a30eb022e5364b0fb4
paper_authors.name: Weiming Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T07:34:49.769Z
paper_authors._id: 64405a9d518271b0d1beea38
paper_authors.avatarUrl: /avatars/b702474588fd7090773320422417a582.svg
paper_authors.fullname: Weiming Ren
paper_authors.type: user
paper_authors.user: wren93
paper_authors._id: 665e92a30eb022e5364b0fb5
paper_authors.name: Aaran Arulraj
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T07:35:11.005Z
paper_authors._id: 65fb7212fc9132a2dfb80061
paper_authors.avatarUrl: /avatars/7955a4c96cba4e5b25481943f7b84be1.svg
paper_authors.fullname: Aaran Arulraj
paper_authors.type: user
paper_authors.user: AaranArulraj
paper_authors._id: 665e92a30eb022e5364b0fb6
paper_authors.name: Xuan He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T07:35:50.254Z
paper_authors._id: 655c3953ce055ed40a7de0ba
paper_authors.avatarUrl: /avatars/1a0a746130fd9b95e20184f2fcf4c0ba.svg
paper_authors.fullname: Xuan He
paper_authors.type: user
paper_authors.user: hexuan21
paper_authors._id: 665e92a30eb022e5364b0fb7
paper_authors.name: Ziyan Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-24T07:33:35.456Z
paper_authors._id: 64778fb8168cb428e00f69b0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64778fb8168cb428e00f69b0/D_XYg74zHG9K3HUJj_gD4.jpeg
paper_authors.fullname: Ziyan Jiang
paper_authors.type: user
paper_authors.user: ziyjiang
paper_authors._id: 665e92a30eb022e5364b0fb8
paper_authors.name: Tianle Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T07:36:51.642Z
paper_authors._id: 63daf95a9f2687298a110386
paper_authors.avatarUrl: /avatars/7e922ca76689de8e0e0c4af350115246.svg
paper_authors.fullname: Tianle LI
paper_authors.type: user
paper_authors.user: tianleliphoebe
paper_authors._id: 665e92a30eb022e5364b0fb9
paper_authors.name: Max Ku
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-04T07:31:33.369Z
paper_authors._id: 631d760344503b7227837242
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631d760344503b7227837242/3b6JRusFX6GKJpsN9ZdeJ.png
paper_authors.fullname: Max Ku
paper_authors.type: user
paper_authors.user: vinesmsuic
paper_authors._id: 665e92a30eb022e5364b0fba
paper_authors.name: Kai Wang
paper_authors._id: 665e92a30eb022e5364b0fbb
paper_authors.name: Alex Zhuang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T07:52:15.887Z
paper_authors._id: 6498eecc3756a91498f08f60
paper_authors.avatarUrl: /avatars/d92a98535d212c73b9249aa48c717f5e.svg
paper_authors.fullname: Zhuang
paper_authors.type: user
paper_authors.user: AlexZhuang
paper_authors._id: 665e92a30eb022e5364b0fbc
paper_authors.name: Rongqi Fan
paper_authors._id: 665e92a30eb022e5364b0fbd
paper_authors.name: Xiang Yue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T07:52:37.143Z
paper_authors._id: 6230d750d93e84e233882dbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
paper_authors.fullname: Xiang Yue
paper_authors.type: user
paper_authors.user: yuexiang96
paper_authors._id: 665e92a30eb022e5364b0fbe
paper_authors.name: Wenhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T07:52:43.625Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_title: MMLU-Pro: A More Robust and Challenging Multi-Task Language
    Understanding Benchmark
paper_summary: In the age of large-scale language models, benchmarks like the Massive
  Multitask Language Understanding (MMLU) have been pivotal in pushing the
  boundaries of what AI can achieve in language comprehension and reasoning
  across diverse domains. However, as models continue to improve, their
  performance on these benchmarks has begun to plateau, making it increasingly
  difficult to discern differences in model capabilities. This paper introduces
  MMLU-Pro, an enhanced dataset designed to extend the mostly knowledge-driven
  MMLU benchmark by integrating more challenging, reasoning-focused questions and
  expanding the choice set from four to ten options. Additionally, MMLU-Pro
  eliminates the trivial and noisy questions in MMLU. Our experimental results
  show that MMLU-Pro not only raises the challenge, causing a significant drop in
  accuracy by 16% to 33% compared to MMLU but also demonstrates greater stability
  under varying prompts. With 24 different prompt styles tested, the sensitivity
  of model scores to prompt variations decreased from 4-5% in MMLU to just 2% in
  MMLU-Pro. Additionally, we found that models utilizing Chain of Thought (CoT)
  reasoning achieved better performance on MMLU-Pro compared to direct answering,
  which is in stark contrast to the findings on the original MMLU, indicating
  that MMLU-Pro includes more complex reasoning questions. Our assessments
  confirm that MMLU-Pro is a more discriminative benchmark to better track
  progress in the field.
paper_ai_keywords: Chain of Thought (CoT) reasoning