title: The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks
summary: As large language models (LLMs) continue to advance in linguistic
  capabilities, robust multilingual evaluation has become essential for promoting
  equitable technological progress. This position paper examines over 2,000
  multilingual (non-English) benchmarks from 148 countries, published between
  2021 and 2024, to evaluate past, present, and future practices in multilingual
  benchmarking. Our findings reveal that, despite significant investments
  amounting to tens of millions of dollars, English remains significantly
  overrepresented in these benchmarks. Additionally, most benchmarks rely on
  original language content rather than translations, with the majority sourced
  from high-resource countries such as China, India, Germany, the UK, and the
  USA. Furthermore, a comparison of benchmark performance with human judgments
  highlights notable disparities. STEM-related tasks exhibit strong correlations
  with human evaluations (0.70 to 0.85), while traditional NLP tasks like
  question answering (e.g., XQuAD) show much weaker correlations (0.11 to 0.30).
  Moreover, translating English benchmarks into other languages proves
  insufficient, as localized benchmarks demonstrate significantly higher
  alignment with local human judgments (0.68) than their translated counterparts
  (0.47). This underscores the importance of creating culturally and
  linguistically tailored benchmarks rather than relying solely on translations.
  Through this comprehensive analysis, we highlight six key limitations in
  current multilingual evaluation practices, propose the guiding principles
  accordingly for effective multilingual benchmarking, and outline five critical
  research directions to drive progress in the field. Finally, we call for a
  global collaborative effort to develop human-aligned benchmarks that prioritize
  real-world applications.
paper_authors._id: 6808458f07e80b69b2df2440
paper_authors.name: Minghao Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:03:02.862Z
paper_authors._id: 62d4bf8c97ab9eb08762a975
paper_authors.avatarUrl: /avatars/73c6228e317cf37b4e3c3e7a4b3d8ae8.svg
paper_authors.fullname: Minghao Wu
paper_authors.type: user
paper_authors.user: minghaowu
paper_authors._id: 6808458f07e80b69b2df2441
paper_authors.name: Weixuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:03:08.988Z
paper_authors._id: 675c52ecdea6ceb2f6ce0ea3
paper_authors.avatarUrl: /avatars/b5ea0e2bc5b001446f62343a907e95f1.svg
paper_authors.fullname: weixuan wang
paper_authors.type: user
paper_authors.user: yourdadishere
paper_authors._id: 6808458f07e80b69b2df2442
paper_authors.name: Sinuo Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:03:19.254Z
paper_authors._id: 635504620d6e89270d440050
paper_authors.avatarUrl: /avatars/3790bf4a68f943a122af59b1362b07f2.svg
paper_authors.fullname: LiuSinuo
paper_authors.type: user
paper_authors.user: SNF
paper_authors._id: 6808458f07e80b69b2df2443
paper_authors.name: Huifeng Yin
paper_authors._id: 6808458f07e80b69b2df2444
paper_authors.name: Xintong Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-12T06:51:30.134Z
paper_authors._id: 674879885d97f2d66a14006a
paper_authors.avatarUrl: /avatars/cdeff44a898560cb460223f30922b081.svg
paper_authors.fullname: Xintong Wang
paper_authors.type: user
paper_authors.user: shanewang
paper_authors._id: 6808458f07e80b69b2df2445
paper_authors.name: Yu Zhao
paper_authors._id: 6808458f07e80b69b2df2446
paper_authors.name: Chenyang Lyu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-23T08:28:16.770Z
paper_authors._id: 6527d8b077bceabaab382a75
paper_authors.avatarUrl: /avatars/69caacf9153dbf6a3796693a968b363f.svg
paper_authors.fullname: Chenyang Lyu
paper_authors.type: user
paper_authors.user: ChenyangLyu
paper_authors._id: 6808458f07e80b69b2df2447
paper_authors.name: Longyue Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:04:14.454Z
paper_authors._id: 636b030c328133bdb3a523bc
paper_authors.avatarUrl: /avatars/15d5d5403fef2f1368bb4185b199061d.svg
paper_authors.fullname: Longyue Wang
paper_authors.type: user
paper_authors.user: longyuewang
paper_authors._id: 6808458f07e80b69b2df2448
paper_authors.name: Weihua Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:04:20.964Z
paper_authors._id: 66b03cedd59c09785e39711e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/N5yfQBSP3oAKPCz4ylR09.png
paper_authors.fullname: Weihua Luo
paper_authors.type: user
paper_authors.user: acecamel1977
paper_authors._id: 6808458f07e80b69b2df2449
paper_authors.name: Kaifu Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:04:27.022Z
paper_authors._id: 63f87ebadf053017d1acbfdd
paper_authors.avatarUrl: /avatars/e497ba5f41a2587837b4a6118d9367bb.svg
paper_authors.fullname: Kaifu Zhang
paper_authors.type: user
paper_authors.user: zhangkaifu314
paper_title: The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks
paper_summary: As large language models (LLMs) continue to advance in linguistic
  capabilities, robust multilingual evaluation has become essential for promoting
  equitable technological progress. This position paper examines over 2,000
  multilingual (non-English) benchmarks from 148 countries, published between
  2021 and 2024, to evaluate past, present, and future practices in multilingual
  benchmarking. Our findings reveal that, despite significant investments
  amounting to tens of millions of dollars, English remains significantly
  overrepresented in these benchmarks. Additionally, most benchmarks rely on
  original language content rather than translations, with the majority sourced
  from high-resource countries such as China, India, Germany, the UK, and the
  USA. Furthermore, a comparison of benchmark performance with human judgments
  highlights notable disparities. STEM-related tasks exhibit strong correlations
  with human evaluations (0.70 to 0.85), while traditional NLP tasks like
  question answering (e.g., XQuAD) show much weaker correlations (0.11 to 0.30).
  Moreover, translating English benchmarks into other languages proves
  insufficient, as localized benchmarks demonstrate significantly higher
  alignment with local human judgments (0.68) than their translated counterparts
  (0.47). This underscores the importance of creating culturally and
  linguistically tailored benchmarks rather than relying solely on translations.
  Through this comprehensive analysis, we highlight six key limitations in
  current multilingual evaluation practices, propose the guiding principles
  accordingly for effective multilingual benchmarking, and outline five critical
  research directions to drive progress in the field. Finally, we call for a
  global collaborative effort to develop human-aligned benchmarks that prioritize
  real-world applications.
paper_ai_summary: Research reveals significant disparities in multilingual benchmark evaluations, emphasizing the need for culturally and linguistically tailored benchmarks over translations to achieve equitable technological progress.