title: Are We Done with MMLU?
summary: Maybe not. We identify and analyse errors in the popular Massive Multitask
  Language Understanding (MMLU) benchmark. Even though MMLU is widely adopted,
  our analysis demonstrates numerous ground truth errors that obscure the true
  capabilities of LLMs. For example, we find that 57% of the analysed questions
  in the Virology subset contain errors. To address this issue, we introduce a
  comprehensive framework for identifying dataset errors using a novel error
  taxonomy. Then, we create MMLU-Redux, which is a subset of 3,000 manually
  re-annotated questions across 30 MMLU subjects. Using MMLU-Redux, we
  demonstrate significant discrepancies with the model performance metrics that
  were originally reported. Our results strongly advocate for revising MMLU's
  error-ridden questions to enhance its future utility and reliability as a
  benchmark. Therefore, we open up MMLU-Redux for additional annotation
  https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux.
paper_authors._id: 66636e5e0b562c29c5bdda45
paper_authors.name: Aryo Pradipta Gema
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T19:54:02.123Z
paper_authors._id: 644f895e23d7eb05ca695054
paper_authors.avatarUrl: /avatars/3fb04dd8544b403262bf98507de05453.svg
paper_authors.fullname: Aryo Pradipta Gema
paper_authors.type: user
paper_authors.user: aryopg
paper_authors._id: 66636e5e0b562c29c5bdda46
paper_authors.name: Joshua Ong Jun Leang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T19:54:08.747Z
paper_authors._id: 6537dd20aa214745cbc89ad9
paper_authors.avatarUrl: /avatars/cb6bda58ae6063562eb02cf92c0e7c4e.svg
paper_authors.fullname: Joshua Ong Jun Leang
paper_authors.type: user
paper_authors.user: Jforeverss
paper_authors._id: 66636e5e0b562c29c5bdda47
paper_authors.name: Giwon Hong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T19:54:14.275Z
paper_authors._id: 653c1a31afb09482c62df54f
paper_authors.avatarUrl: /avatars/e11b00995fd32e421019018a90359eb0.svg
paper_authors.fullname: Giwon Hong
paper_authors.type: user
paper_authors.user: GWHed
paper_authors._id: 66636e5e0b562c29c5bdda48
paper_authors.name: Alessio Devoto
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T19:54:20.984Z
paper_authors._id: 62180288d05d07ebf7de5f31
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62180288d05d07ebf7de5f31/io2RsaRSjIDktQylXVvEB.jpeg
paper_authors.fullname: Alessio
paper_authors.type: user
paper_authors.user: alessiodevoto
paper_authors._id: 66636e5e0b562c29c5bdda49
paper_authors.name: Alberto Carlo Maria Mancino
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T19:54:27.659Z
paper_authors._id: 663a42b49b8151660f4c4cba
paper_authors.avatarUrl: /avatars/f53b2da495386df834fbc7e4c35eee1d.svg
paper_authors.fullname: Alberto Carlo Maria Mancino
paper_authors.type: user
paper_authors.user: albertomancino
paper_authors._id: 66636e5e0b562c29c5bdda4a
paper_authors.name: Rohit Saxena
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T13:46:33.918Z
paper_authors._id: 657ccbf2869d5bb0e53b482f
paper_authors.avatarUrl: /avatars/2eae5a10bdc14814a04d9f255f16de6b.svg
paper_authors.fullname: Rohit Saxena
paper_authors.type: user
paper_authors.user: rohitsaxena
paper_authors._id: 66636e5e0b562c29c5bdda4b
paper_authors.name: Xuanli He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T19:54:33.526Z
paper_authors._id: 614b0ab851b6682b9e56fd92
paper_authors.avatarUrl: /avatars/81606ae85106b89a34bd08283646d07b.svg
paper_authors.fullname: Xuanli He
paper_authors.type: user
paper_authors.user: zodiache
paper_authors._id: 66636e5e0b562c29c5bdda4c
paper_authors.name: Yu Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-09T20:04:38.695Z
paper_authors._id: 654a500d9b8bd6406d431c0d
paper_authors.avatarUrl: /avatars/a6b76441bbc6f4b71d49c52e454c9ef7.svg
paper_authors.fullname: Yu Zhao
paper_authors.type: user
paper_authors.user: yuzhaouoe
paper_authors._id: 66636e5e0b562c29c5bdda4d
paper_authors.name: Xiaotang Du
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:46:56.843Z
paper_authors._id: 63d4698d1da8c28b7b8ab04a
paper_authors.avatarUrl: /avatars/73d4f49018db03a59e72b69bb2d1dd0c.svg
paper_authors.fullname: Xiaotang Du
paper_authors.type: user
paper_authors.user: acDante
paper_authors._id: 66636e5e0b562c29c5bdda4e
paper_authors.name: Mohammad Reza Ghasemi Madani
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T12:00:24.500Z
paper_authors._id: 63d03e196b985b0f25c91bcc
paper_authors.avatarUrl: /avatars/974fa4ed6f368602436a305377365152.svg
paper_authors.fullname: Mohammad Reza Ghasemi Madani
paper_authors.type: user
paper_authors.user: Reza-Madani
paper_authors._id: 66636e5e0b562c29c5bdda4f
paper_authors.name: Claire Barale
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T19:54:39.624Z
paper_authors._id: 652d6dd40517d2c3746fb73a
paper_authors.avatarUrl: /avatars/9c8b5167b98a33a5f85aa1a6eb50eb36.svg
paper_authors.fullname: Claire Barale
paper_authors.type: user
paper_authors.user: clairebarale
paper_authors._id: 66636e5e0b562c29c5bdda50
paper_authors.name: Robert McHardy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T19:54:57.806Z
paper_authors._id: 64b91a3fa62c52b252cf39b2
paper_authors.avatarUrl: /avatars/af80d43464d8a8bacdb32c5b77d1b001.svg
paper_authors.fullname: Robert McHardy 
paper_authors.type: user
paper_authors.user: robmchinst
paper_authors._id: 66636e5e0b562c29c5bdda51
paper_authors.name: Joshua Harris
paper_authors._id: 66636e5e0b562c29c5bdda52
paper_authors.name: Jean Kaddour
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T19:55:17.616Z
paper_authors._id: 6304061c0547362a22a76a17
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661339692442-6304061c0547362a22a76a17.jpeg
paper_authors.fullname: Jean Kaddour
paper_authors.type: user
paper_authors.user: JeanKaddour
paper_authors._id: 66636e5e0b562c29c5bdda53
paper_authors.name: Emile van Krieken
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T12:00:23.118Z
paper_authors._id: 62cd4ae83e5ba89c40f15156
paper_authors.avatarUrl: /avatars/f70aca85f8836edaedcee324a18140b5.svg
paper_authors.fullname: Emile van Krieken
paper_authors.type: user
paper_authors.user: HEmile
paper_authors._id: 66636e5e0b562c29c5bdda54
paper_authors.name: Pasquale Minervini
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T12:00:21.688Z
paper_authors._id: 61001311e043e15c13412d30
paper_authors.avatarUrl: /avatars/eea1e4c39decee282f2940d122090491.svg
paper_authors.fullname: Pasquale Minervini
paper_authors.type: user
paper_authors.user: pminervini
paper_title: Are We Done with MMLU?
paper_summary: Maybe not. We identify and analyse errors in the popular Massive Multitask
  Language Understanding (MMLU) benchmark. Even though MMLU is widely adopted,
  our analysis demonstrates numerous ground truth errors that obscure the true
  capabilities of LLMs. For example, we find that 57% of the analysed questions
  in the Virology subset contain errors. To address this issue, we introduce a
  comprehensive framework for identifying dataset errors using a novel error
  taxonomy. Then, we create MMLU-Redux, which is a subset of 3,000 manually
  re-annotated questions across 30 MMLU subjects. Using MMLU-Redux, we
  demonstrate significant discrepancies with the model performance metrics that
  were originally reported. Our results strongly advocate for revising MMLU's
  error-ridden questions to enhance its future utility and reliability as a
  benchmark. Therefore, we open up MMLU-Redux for additional annotation
  https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux.
paper_ai_keywords: LRM muffin MMLU Virology subset ground truth errors weight truth errors self-annotation MMLU-Redux generative model performance metrics ASI MMLU-Redux