title: Global MMLU: Understanding and Addressing Cultural and Linguistic Biases
    in Multilingual Evaluation
summary: Cultural biases in multilingual datasets pose significant challenges for
  their effectiveness as global benchmarks. These biases stem not only from
  language but also from the cultural knowledge required to interpret questions,
  reducing the practical utility of translated datasets like MMLU. Furthermore,
  translation often introduces artifacts that can distort the meaning or clarity
  of questions in the target language. A common practice in multilingual
  evaluation is to rely on machine-translated evaluation sets, but simply
  translating a dataset is insufficient to address these challenges. In this
  work, we trace the impact of both of these issues on multilingual evaluations
  and ensuing model performances. Our large-scale evaluation of state-of-the-art
  open and proprietary models illustrates that progress on MMLU depends heavily
  on learning Western-centric concepts, with 28% of all questions requiring
  culturally sensitive knowledge. Moreover, for questions requiring geographic
  knowledge, an astounding 84.9% focus on either North American or European
  regions. Rankings of model evaluations change depending on whether they are
  evaluated on the full portion or the subset of questions annotated as
  culturally sensitive, showing the distortion to model rankings when blindly
  relying on translated MMLU. We release Global-MMLU, an improved MMLU with
  evaluation coverage across 42 languages -- with improved overall quality by
  engaging with compensated professional and community annotators to verify
  translation quality while also rigorously evaluating cultural biases present in
  the original dataset. This comprehensive Global-MMLU set also includes
  designated subsets labeled as culturally sensitive and culturally agnostic to
  allow for more holistic, complete evaluation.
paper_authors._id: 675193c76af9371edd506f32
paper_authors.name: Shivalika Singh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T15:57:32.994Z
paper_authors._id: 62c2175d756039dd0dd20509
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677753847837-62c2175d756039dd0dd20509.jpeg
paper_authors.fullname: Shivalika Singh
paper_authors.type: user
paper_authors.user: shivalikasingh
paper_authors._id: 675193c76af9371edd506f33
paper_authors.name: Angelika Romanou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T15:57:21.884Z
paper_authors._id: 62a0803e3e7d1dda046d475a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a0803e3e7d1dda046d475a/Oi1qHSaX7_PU1pS8qaXKv.jpeg
paper_authors.fullname: Angelika Romanou
paper_authors.type: user
paper_authors.user: angelika
paper_authors._id: 675193c76af9371edd506f34
paper_authors.name: Clémentine Fourrier
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T17:14:27.962Z
paper_authors._id: 6202a599216215a22221dea9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1644340617257-noauth.png
paper_authors.fullname: Clémentine Fourrier
paper_authors.type: user
paper_authors.user: clefourrier
paper_authors._id: 675193c76af9371edd506f35
paper_authors.name: David I. Adelani
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T15:57:24.007Z
paper_authors._id: 5fcc1929563427b03e9af259
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1620642852901-5fcc1929563427b03e9af259.jpeg
paper_authors.fullname: David Adelani
paper_authors.type: user
paper_authors.user: Davlan
paper_authors._id: 675193c76af9371edd506f36
paper_authors.name: Jian Gang Ngui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T15:57:17.762Z
paper_authors._id: 65727254948b9e1dbc1eb717
paper_authors.avatarUrl: /avatars/af0c1d4c5ad16f7c856538dd7c276cf6.svg
paper_authors.fullname: Jian Gang Ngui
paper_authors.type: user
paper_authors.user: Jian-Gang
paper_authors._id: 675193c76af9371edd506f37
paper_authors.name: Daniel Vila-Suero
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T15:57:35.570Z
paper_authors._id: 60420dccc15e823a685f2b03
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60420dccc15e823a685f2b03/Dn7QTyy9SZ7jKN6xpufVD.png
paper_authors.fullname: Daniel Vila
paper_authors.type: user
paper_authors.user: dvilasuero
paper_authors._id: 675193c76af9371edd506f38
paper_authors.name: Peerat Limkonchotiwat
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T11:07:23.442Z
paper_authors._id: 612cc140ed7fc620ce36dbe0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1633414888277-612cc140ed7fc620ce36dbe0.jpeg
paper_authors.fullname: Peerat Limkonchotiwat
paper_authors.type: user
paper_authors.user: mrp
paper_authors._id: 675193c76af9371edd506f39
paper_authors.name: Kelly Marchisio
paper_authors._id: 675193c76af9371edd506f3a
paper_authors.name: Wei Qi Leong
paper_authors._id: 675193c76af9371edd506f3b
paper_authors.name: Yosephine Susanto
paper_authors._id: 675193c76af9371edd506f3c
paper_authors.name: Raymond Ng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T11:07:25.659Z
paper_authors._id: 63437afe5b97f5093890e38e
paper_authors.avatarUrl: /avatars/0c1a6f2f9a28d74fc641f10a1bf2867c.svg
paper_authors.fullname: Raymond Ng
paper_authors.type: user
paper_authors.user: RaymondAISG
paper_authors._id: 675193c76af9371edd506f3d
paper_authors.name: Shayne Longpre
paper_authors._id: 675193c76af9371edd506f3e
paper_authors.name: Wei-Yin Ko
paper_authors._id: 675193c76af9371edd506f3f
paper_authors.name: Madeline Smith
paper_authors._id: 675193c76af9371edd506f40
paper_authors.name: Antoine Bosselut
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-06T13:34:35.555Z
paper_authors._id: 654fb1c67490049d621516fb
paper_authors.avatarUrl: /avatars/0f88177da236b1bce4a4a58fd0246884.svg
paper_authors.fullname: Antoine Bosselut
paper_authors.type: user
paper_authors.user: atcbosselut
paper_authors._id: 675193c76af9371edd506f41
paper_authors.name: Alice Oh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-13T08:54:48.572Z
paper_authors._id: 60e0251ea9b5d8282481f2b7
paper_authors.avatarUrl: /avatars/43441373af054a6184c22097bfeb97e4.svg
paper_authors.fullname: Alice Oh
paper_authors.type: user
paper_authors.user: aliceoh
paper_authors._id: 675193c76af9371edd506f42
paper_authors.name: Andre F. T. Martins
paper_authors._id: 675193c76af9371edd506f43
paper_authors.name: Leshem Choshen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T15:57:19.561Z
paper_authors._id: 61bf40824b4300d0fb0acf59
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1644224872623-61bf40824b4300d0fb0acf59.jpeg
paper_authors.fullname: Leshem Choshen
paper_authors.type: user
paper_authors.user: borgr
paper_authors._id: 675193c76af9371edd506f44
paper_authors.name: Daphne Ippolito
paper_authors._id: 675193c76af9371edd506f45
paper_authors.name: Enzo Ferrante
paper_authors._id: 675193c76af9371edd506f46
paper_authors.name: Marzieh Fadaee
paper_authors._id: 675193c76af9371edd506f47
paper_authors.name: Beyza Ermis
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T11:07:27.564Z
paper_authors._id: 6634e8c750ea73bfbc59223e
paper_authors.avatarUrl: /avatars/9f0a7bcdfd82d5344932437d458fce9f.svg
paper_authors.fullname: Beyza Ermis
paper_authors.type: user
paper_authors.user: beyzaermis
paper_authors._id: 675193c76af9371edd506f48
paper_authors.name: Sara Hooker
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-20T09:30:46.602Z
paper_authors._id: 63434eb76f59b79da07dbddf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63434eb76f59b79da07dbddf/BEwmVjqPNYlqmutXG0G6e.jpeg
paper_authors.fullname: Sara Hooker
paper_authors.type: user
paper_authors.user: sarahooker
paper_title: Global MMLU: Understanding and Addressing Cultural and Linguistic Biases
    in Multilingual Evaluation
paper_summary: Cultural biases in multilingual datasets pose significant challenges for
  their effectiveness as global benchmarks. These biases stem not only from
  language but also from the cultural knowledge required to interpret questions,
  reducing the practical utility of translated datasets like MMLU. Furthermore,
  translation often introduces artifacts that can distort the meaning or clarity
  of questions in the target language. A common practice in multilingual
  evaluation is to rely on machine-translated evaluation sets, but simply
  translating a dataset is insufficient to address these challenges. In this
  work, we trace the impact of both of these issues on multilingual evaluations
  and ensuing model performances. Our large-scale evaluation of state-of-the-art
  open and proprietary models illustrates that progress on MMLU depends heavily
  on learning Western-centric concepts, with 28% of all questions requiring
  culturally sensitive knowledge. Moreover, for questions requiring geographic
  knowledge, an astounding 84.9% focus on either North American or European
  regions. Rankings of model evaluations change depending on whether they are
  evaluated on the full portion or the subset of questions annotated as
  culturally sensitive, showing the distortion to model rankings when blindly
  relying on translated MMLU. We release Global-MMLU, an improved MMLU with
  evaluation coverage across 42 languages -- with improved overall quality by
  engaging with compensated professional and community annotators to verify
  translation quality while also rigorously evaluating cultural biases present in
  the original dataset. This comprehensive Global-MMLU set also includes
  designated subsets labeled as culturally sensitive and culturally agnostic to
  allow for more holistic, complete evaluation.
paper_ai_keywords: multilingual datasets MMLU machine-translated evaluation sets multilingual evaluations state-of-the-art open and proprietary models culturally sensitive knowledge geographic knowledge North American European model rankings Global-MMLU professional and community annotators cultural biases