Global MMLU: Understanding and Addressing Cultural and Linguistic Biases
  in Multilingual Evaluation
Cultural biases in multilingual datasets pose significant challenges for
their effectiveness as global benchmarks. These biases stem not only from
language but also from the cultural knowledge required to interpret questions,
reducing the practical utility of translated datasets like MMLU. Furthermore,
translation often introduces artifacts that can distort the meaning or clarity
of questions in the target language. A common practice in multilingual
evaluation is to rely on machine-translated evaluation sets, but simply
translating a dataset is insufficient to address these challenges. In this
work, we trace the impact of both of these issues on multilingual evaluations
and ensuing model performances. Our large-scale evaluation of state-of-the-art
open and proprietary models illustrates that progress on MMLU depends heavily
on learning Western-centric concepts, with 28% of all questions requiring
culturally sensitive knowledge. Moreover, for questions requiring geographic
knowledge, an astounding 84.9% focus on either North American or European
regions. Rankings of model evaluations change depending on whether they are
evaluated on the full portion or the subset of questions annotated as
culturally sensitive, showing the distortion to model rankings when blindly
relying on translated MMLU. We release Global-MMLU, an improved MMLU with
evaluation coverage across 42 languages -- with improved overall quality by
engaging with compensated professional and community annotators to verify
translation quality while also rigorously evaluating cultural biases present in
the original dataset. This comprehensive Global-MMLU set also includes
designated subsets labeled as culturally sensitive and culturally agnostic to
allow for more holistic, complete evaluation.
675193c76af9371edd506f32
Shivalika Singh
claimed_verified
2024-12-06T15:57:32.994Z
62c2175d756039dd0dd20509
https://cdn-avatars.huggingface.co/v1/production/uploads/1677753847837-62c2175d756039dd0dd20509.jpeg
Shivalika Singh
user
shivalikasingh
675193c76af9371edd506f33
Angelika Romanou
claimed_verified
2024-12-06T15:57:21.884Z
62a0803e3e7d1dda046d475a
https://cdn-avatars.huggingface.co/v1/production/uploads/62a0803e3e7d1dda046d475a/Oi1qHSaX7_PU1pS8qaXKv.jpeg
Angelika Romanou
user
angelika
675193c76af9371edd506f34
Clémentine Fourrier
claimed_verified
2024-12-06T17:14:27.962Z
6202a599216215a22221dea9
https://cdn-avatars.huggingface.co/v1/production/uploads/1644340617257-noauth.png
Clémentine Fourrier
user
clefourrier
675193c76af9371edd506f35
David I. Adelani
claimed_verified
2024-12-06T15:57:24.007Z
5fcc1929563427b03e9af259
https://cdn-avatars.huggingface.co/v1/production/uploads/1620642852901-5fcc1929563427b03e9af259.jpeg
David Adelani
user
Davlan
675193c76af9371edd506f36
Jian Gang Ngui
claimed_verified
2024-12-06T15:57:17.762Z
65727254948b9e1dbc1eb717
/avatars/af0c1d4c5ad16f7c856538dd7c276cf6.svg
Jian Gang Ngui
user
Jian-Gang
675193c76af9371edd506f37
Daniel Vila-Suero
claimed_verified
2024-12-06T15:57:35.570Z
60420dccc15e823a685f2b03
https://cdn-avatars.huggingface.co/v1/production/uploads/60420dccc15e823a685f2b03/AOApMWt_jvm9e6XQ2vlrJ.jpeg
Daniel Vila
user
dvilasuero
675193c76af9371edd506f38
Peerat Limkonchotiwat
claimed_verified
2024-12-09T11:07:23.442Z
612cc140ed7fc620ce36dbe0
https://cdn-avatars.huggingface.co/v1/production/uploads/1633414888277-612cc140ed7fc620ce36dbe0.jpeg
Peerat Limkonchotiwat
user
mrp
675193c76af9371edd506f39
Kelly Marchisio
claimed_verified
2025-05-16T07:13:21.183Z
66356b1e2fcf64c1548a1233
/avatars/9a1fab7da6cd16e8bf2d104c399ae34f.svg
Kelly Marchisio
user
kellycohere
675193c76af9371edd506f3a
Wei Qi Leong
675193c76af9371edd506f3b
Yosephine Susanto
675193c76af9371edd506f3c
Raymond Ng
claimed_verified
2024-12-09T11:07:25.659Z
63437afe5b97f5093890e38e
/avatars/0c1a6f2f9a28d74fc641f10a1bf2867c.svg
Raymond Ng
user
RaymondAISG
675193c76af9371edd506f3d
Shayne Longpre
675193c76af9371edd506f3e
Wei-Yin Ko
675193c76af9371edd506f3f
Madeline Smith
675193c76af9371edd506f40
Antoine Bosselut
claimed_verified
2025-03-06T13:34:35.555Z
654fb1c67490049d621516fb
/avatars/0f88177da236b1bce4a4a58fd0246884.svg
Antoine Bosselut
user
atcbosselut
675193c76af9371edd506f41
Alice Oh
claimed_verified
2025-01-13T08:54:48.572Z
60e0251ea9b5d8282481f2b7
/avatars/43441373af054a6184c22097bfeb97e4.svg
Alice Oh
user
aliceoh
675193c76af9371edd506f42
Andre F. T. Martins
675193c76af9371edd506f43
Leshem Choshen
claimed_verified
2024-12-06T15:57:19.561Z
61bf40824b4300d0fb0acf59
https://cdn-avatars.huggingface.co/v1/production/uploads/1644224872623-61bf40824b4300d0fb0acf59.jpeg
Leshem Choshen
user
borgr
675193c76af9371edd506f44
Daphne Ippolito
675193c76af9371edd506f45
Enzo Ferrante
675193c76af9371edd506f46
Marzieh Fadaee
675193c76af9371edd506f47
Beyza Ermis
claimed_verified
2024-12-09T11:07:27.564Z
6634e8c750ea73bfbc59223e
/avatars/9f0a7bcdfd82d5344932437d458fce9f.svg
Beyza Ermis
user
beyzaermis
675193c76af9371edd506f48
Sara Hooker
claimed_verified
2025-01-20T09:30:46.602Z
63434eb76f59b79da07dbddf
https://cdn-avatars.huggingface.co/v1/production/uploads/63434eb76f59b79da07dbddf/BEwmVjqPNYlqmutXG0G6e.jpeg
Sara Hooker
user
sarahooker
Global MMLU: Understanding and Addressing Cultural and Linguistic Biases
  in Multilingual Evaluation
Cultural biases in multilingual datasets pose significant challenges for
their effectiveness as global benchmarks. These biases stem not only from
language but also from the cultural knowledge required to interpret questions,
reducing the practical utility of translated datasets like MMLU. Furthermore,
translation often introduces artifacts that can distort the meaning or clarity
of questions in the target language. A common practice in multilingual
evaluation is to rely on machine-translated evaluation sets, but simply
translating a dataset is insufficient to address these challenges. In this
work, we trace the impact of both of these issues on multilingual evaluations
and ensuing model performances. Our large-scale evaluation of state-of-the-art
open and proprietary models illustrates that progress on MMLU depends heavily
on learning Western-centric concepts, with 28% of all questions requiring
culturally sensitive knowledge. Moreover, for questions requiring geographic
knowledge, an astounding 84.9% focus on either North American or European
regions. Rankings of model evaluations change depending on whether they are
evaluated on the full portion or the subset of questions annotated as
culturally sensitive, showing the distortion to model rankings when blindly
relying on translated MMLU. We release Global-MMLU, an improved MMLU with
evaluation coverage across 42 languages -- with improved overall quality by
engaging with compensated professional and community annotators to verify
translation quality while also rigorously evaluating cultural biases present in
the original dataset. This comprehensive Global-MMLU set also includes
designated subsets labeled as culturally sensitive and culturally agnostic to
allow for more holistic, complete evaluation.
Global-MMLU addresses cultural biases and translation artifacts in multilingual datasets by involving professional annotators and offering culturally distinct subsets for more accurate model evaluation.
cultural biases multilingual datasets MMLU machine-translated evaluation sets model evaluations culturally sensitive geographic knowledge North America Europe Global-MMLU professional annotators community annotators translation quality