INCLUDE: Evaluating Multilingual Language Understanding with Regional
  Knowledge
The performance differential of large language models (LLM) between languages
hinders their effective deployment in many regions, inhibiting the potential
economic and societal value of generative AI tools in many communities.
However, the development of functional LLMs in many languages (\ie,
multilingual LLMs) is bottlenecked by the lack of high-quality evaluation
resources in languages other than English. Moreover, current practices in
multilingual benchmark construction often translate English resources, ignoring
the regional and cultural knowledge of the environments in which multilingual
systems would be used. In this work, we construct an evaluation suite of
197,243 QA pairs from local exam sources to measure the capabilities of
multilingual LLMs in a variety of regional contexts. Our novel resource,
INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across
44 written languages that evaluates multilingual LLMs for performance in the
actual language environments where they would be deployed.
674d75a01e1ca74a41c21237
Angelika Romanou
extracted_confirmed
2024-12-02T09:05:32.078Z
62a0803e3e7d1dda046d475a
https://cdn-avatars.huggingface.co/v1/production/uploads/62a0803e3e7d1dda046d475a/Oi1qHSaX7_PU1pS8qaXKv.jpeg
Angelika Romanou
user
angelika
674d75a01e1ca74a41c21238
Negar Foroutan
674d75a01e1ca74a41c21239
Anna Sotnikova
674d75a01e1ca74a41c2123a
Zeming Chen
claimed_verified
2024-12-04T11:52:39.175Z
654b92086a49f6f6e0edbcd9
https://cdn-avatars.huggingface.co/v1/production/uploads/654b92086a49f6f6e0edbcd9/1bWswk7EcL04mWwgx1TpA.jpeg
Zeming Chen - Eric
user
zechen-nlp
674d75a01e1ca74a41c2123b
Sree Harsha Nelaturu
claimed_verified
2025-05-12T12:56:33.493Z
6238f87f35384c2bcccb3889
/avatars/80e19fae8bc194c8018860a908833893.svg
Sree Harsha Nelaturu
user
deepmage121
674d75a01e1ca74a41c2123c
Shivalika Singh
claimed_verified
2024-12-06T15:57:41.933Z
62c2175d756039dd0dd20509
https://cdn-avatars.huggingface.co/v1/production/uploads/1677753847837-62c2175d756039dd0dd20509.jpeg
Shivalika Singh
user
shivalikasingh
674d75a01e1ca74a41c2123d
Rishabh Maheshwary
claimed_verified
2025-05-26T08:44:52.037Z
645c26d423ed9b7788d5e24b
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/cZMUluWpYUlSLcn6yoC7c.jpeg
Rishabh Maheshwary
user
rmahesh
674d75a01e1ca74a41c2123e
Micol Altomare
674d75a01e1ca74a41c2123f
Mohamed A. Haggag
674d75a01e1ca74a41c21240
Snegha A
674d75a01e1ca74a41c21241
Alfonso Amayuelas
claimed_verified
2025-03-24T08:56:06.728Z
634899b203b5a62a0d1ef9ee
/avatars/f0019a4555c0430aa428940727dc24f1.svg
Alfonso Amayuelas
user
amayuelas
674d75a01e1ca74a41c21242
Azril Hafizi Amirudin
674d75a01e1ca74a41c21243
Viraat Aryabumi
674d75a01e1ca74a41c21244
Danylo Boiko
claimed_verified
2025-02-03T08:16:29.785Z
6655a52cbb6e96974bcbe289
/avatars/7175adff262cb38bce42a10ddee354c3.svg
Danylo Boiko
user
danylo-boiko
674d75a01e1ca74a41c21245
Michael Chang
674d75a01e1ca74a41c21246
Jenny Chim
674d75a01e1ca74a41c21247
Gal Cohen
674d75a01e1ca74a41c21248
Aditya Kumar Dalmia
674d75a01e1ca74a41c21249
Abraham Diress
674d75a01e1ca74a41c2124a
Sharad Duwal
674d75a01e1ca74a41c2124b
Daniil Dzenhaliou
674d75a01e1ca74a41c2124c
Daniel Fernando Erazo Florez
674d75a01e1ca74a41c2124d
Fabian Farestam
claimed_verified
2025-04-01T15:59:52.237Z
649edc8ca74a8d2dc44f117e
/avatars/33064f5dd5c8029278cdaa01cd16d4ab.svg
Fabian Farestam
user
northern-64bit
674d75a01e1ca74a41c2124e
Joseph Marvin Imperial
674d75a01e1ca74a41c2124f
Shayekh Bin Islam
claimed_verified
2024-12-02T17:31:54.464Z
5e4b943a37cb5b49818287b5
https://cdn-avatars.huggingface.co/v1/production/uploads/5e4b943a37cb5b49818287b5/Sd8QOGFY-_B1z4UlFTuPf.jpeg
Shayekh Islam
user
shayekh
674d75a01e1ca74a41c21250
Perttu Isotalo
674d75a01e1ca74a41c21251
Maral Jabbarishiviari
674d75a01e1ca74a41c21252
Börje F. Karlsson
claimed_verified
2025-03-11T08:24:49.499Z
61e52be53d6dbb1da842316a
https://cdn-avatars.huggingface.co/v1/production/uploads/61e52be53d6dbb1da842316a/gx0WGPcOCClXPymoKglc4.jpeg
Börje Karlsson
user
tellarin
674d75a01e1ca74a41c21253
Eldar Khalilov
674d75a01e1ca74a41c21254
Christopher Klamm
674d75a01e1ca74a41c21255
Fajri Koto
674d75a01e1ca74a41c21256
Dominik Krzemiński
claimed_verified
2025-02-24T10:58:18.861Z
662cc845616128914a3c9817
/avatars/cd186de00f28bb866abc1ab6c4465663.svg
DomKrz
user
dokato
674d75a01e1ca74a41c21257
Gabriel Adriano de Melo
674d75a01e1ca74a41c21258
Syrielle Montariol
674d75a01e1ca74a41c21259
Yiyang Nan
claimed_verified
2025-05-15T10:41:21.112Z
66ad9237091a147c4f154dbb
/avatars/c80258e877bafc08452f241a6ecfec04.svg
Yiyang Nan
user
olivernan
674d75a01e1ca74a41c2125a
Joel Niklaus
674d75a01e1ca74a41c2125b
Jekaterina Novikova
claimed_verified
2024-12-30T19:37:20.722Z
6136d5e0e803998e3b48f063
https://cdn-avatars.huggingface.co/v1/production/uploads/6136d5e0e803998e3b48f063/Uitq0a4W9qBvkRNJ9Eiyh.jpeg
Jekaterina Novikova
user
Jekaterina
674d75a01e1ca74a41c2125c
Johan Samir Obando Ceron
674d75a01e1ca74a41c2125d
Debjit Paul
674d75a01e1ca74a41c2125e
Esther Ploeger
674d75a01e1ca74a41c2125f
Jebish Purbey
claimed_verified
2024-12-03T11:16:06.831Z
66c578770a22b2f9ab575847
https://cdn-avatars.huggingface.co/v1/production/uploads/66c578770a22b2f9ab575847/-zfNho1DR3yZHDazq669-.png
Jebish Purbey
user
jebish7
674d75a01e1ca74a41c21260
Swati Rajwal
674d75a01e1ca74a41c21261
Selvan Sunitha Ravi
674d75a01e1ca74a41c21262
Sara Rydell
674d75a01e1ca74a41c21263
Roshan Santhosh
claimed_verified
2024-12-20T08:34:07.639Z
65f5a22e6bd4bac5b6fa7d1d
https://cdn-avatars.huggingface.co/v1/production/uploads/65f5a22e6bd4bac5b6fa7d1d/1-ftEEvA66TfDa-KL1O3x.jpeg
Roshan Santhosh
user
roshansk23
674d75a01e1ca74a41c21264
Drishti Sharma
claimed_verified
2024-12-04T09:46:36.448Z
618c1ad1c74578e0a4a4d074
https://cdn-avatars.huggingface.co/v1/production/uploads/618c1ad1c74578e0a4a4d074/8u_AkeHt4d6xtQ8hzaffU.jpeg
Drishti Sharma
user
DrishtiSharma
674d75a01e1ca74a41c21265
Marjana Prifti Skenduli
claimed_verified
2025-05-08T09:00:27.460Z
662be1838082c634bb916fb7
https://cdn-avatars.huggingface.co/v1/production/uploads/662be1838082c634bb916fb7/0xSOVwZDYTDGhUsLOVvis.jpeg
Marjana Prifti Skenduli
user
marjpri
674d75a01e1ca74a41c21266
Arshia Soltani Moakhar
674d75a01e1ca74a41c21267
Bardia Soltani Moakhar
674d75a01e1ca74a41c21268
Ran Tamir
674d75a01e1ca74a41c21269
Ayush Kumar Tarun
674d75a01e1ca74a41c2126a
Azmine Toushik Wasi
674d75a01e1ca74a41c2126b
Thenuka Ovin Weerasinghe
674d75a01e1ca74a41c2126c
Serhan Yilmaz
claimed_verified
2025-02-28T12:17:04.137Z
64b011ca9d08245fa862e482
https://cdn-avatars.huggingface.co/v1/production/uploads/64b011ca9d08245fa862e482/txKvPNMXNqVSUkrJgr4Yl.jpeg
Serhan YILMAZ
user
serhany
674d75a01e1ca74a41c2126d
Mike Zhang
claimed_verified
2024-12-02T08:56:04.648Z
60d33fbbd7b174177faabd4f
https://cdn-avatars.huggingface.co/v1/production/uploads/60d33fbbd7b174177faabd4f/pfyv_xj2B2m2N4F4sT9zJ.jpeg
Mike Zhang
user
jjzha
674d75a01e1ca74a41c2126e
Imanol Schlag
674d75a01e1ca74a41c2126f
Marzieh Fadaee
claimed_verified
2024-12-03T14:40:43.732Z
6441042d5d600fb0951a5f99
https://cdn-avatars.huggingface.co/v1/production/uploads/6441042d5d600fb0951a5f99/4CbOaYcEz99BtVAQvnGTn.jpeg
Marzieh Fadaee
user
MarziehFadaee
674d75a01e1ca74a41c21270
Sara Hooker
claimed_verified
2025-01-20T09:30:48.624Z
63434eb76f59b79da07dbddf
https://cdn-avatars.huggingface.co/v1/production/uploads/63434eb76f59b79da07dbddf/BEwmVjqPNYlqmutXG0G6e.jpeg
Sara Hooker
user
sarahooker
674d75a01e1ca74a41c21271
Antoine Bosselut
extracted_confirmed
2024-12-02T08:57:36.004Z
654fb1c67490049d621516fb
/avatars/0f88177da236b1bce4a4a58fd0246884.svg
Antoine Bosselut
user
atcbosselut
INCLUDE: Evaluating Multilingual Language Understanding with Regional
  Knowledge
The performance differential of large language models (LLM) between languages
hinders their effective deployment in many regions, inhibiting the potential
economic and societal value of generative AI tools in many communities.
However, the development of functional LLMs in many languages (\ie,
multilingual LLMs) is bottlenecked by the lack of high-quality evaluation
resources in languages other than English. Moreover, current practices in
multilingual benchmark construction often translate English resources, ignoring
the regional and cultural knowledge of the environments in which multilingual
systems would be used. In this work, we construct an evaluation suite of
197,243 QA pairs from local exam sources to measure the capabilities of
multilingual LLMs in a variety of regional contexts. Our novel resource,
INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across
44 written languages that evaluates multilingual LLMs for performance in the
actual language environments where they would be deployed.
INCLUDE, a novel evaluation suite with QA pairs from local exam sources, measures multilingual LLM performance across 44 languages for real-world deployment.
large language models multilingual LLMs evaluation resources multilingual benchmark regional contexts knowledge-centric benchmark reasoning-centric benchmark