title: INCLUDE: Evaluating Multilingual Language Understanding with Regional
    Knowledge
summary: The performance differential of large language models (LLM) between languages
  hinders their effective deployment in many regions, inhibiting the potential
  economic and societal value of generative AI tools in many communities.
  However, the development of functional LLMs in many languages (\ie,
  multilingual LLMs) is bottlenecked by the lack of high-quality evaluation
  resources in languages other than English. Moreover, current practices in
  multilingual benchmark construction often translate English resources, ignoring
  the regional and cultural knowledge of the environments in which multilingual
  systems would be used. In this work, we construct an evaluation suite of
  197,243 QA pairs from local exam sources to measure the capabilities of
  multilingual LLMs in a variety of regional contexts. Our novel resource,
  INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across
  44 written languages that evaluates multilingual LLMs for performance in the
  actual language environments where they would be deployed.
paper_authors._id: 674d75a01e1ca74a41c21237
paper_authors.name: Angelika Romanou
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-12-02T09:05:32.078Z
paper_authors._id: 62a0803e3e7d1dda046d475a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a0803e3e7d1dda046d475a/Oi1qHSaX7_PU1pS8qaXKv.jpeg
paper_authors.fullname: Angelika Romanou
paper_authors.type: user
paper_authors.user: angelika
paper_authors._id: 674d75a01e1ca74a41c21238
paper_authors.name: Negar Foroutan
paper_authors._id: 674d75a01e1ca74a41c21239
paper_authors.name: Anna Sotnikova
paper_authors._id: 674d75a01e1ca74a41c2123a
paper_authors.name: Zeming Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T11:52:39.175Z
paper_authors._id: 654b92086a49f6f6e0edbcd9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/654b92086a49f6f6e0edbcd9/1bWswk7EcL04mWwgx1TpA.jpeg
paper_authors.fullname: Zeming Chen - Eric
paper_authors.type: user
paper_authors.user: zechen-nlp
paper_authors._id: 674d75a01e1ca74a41c2123b
paper_authors.name: Sree Harsha Nelaturu
paper_authors._id: 674d75a01e1ca74a41c2123c
paper_authors.name: Shivalika Singh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T15:57:41.933Z
paper_authors._id: 62c2175d756039dd0dd20509
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677753847837-62c2175d756039dd0dd20509.jpeg
paper_authors.fullname: Shivalika Singh
paper_authors.type: user
paper_authors.user: shivalikasingh
paper_authors._id: 674d75a01e1ca74a41c2123d
paper_authors.name: Rishabh Maheshwary
paper_authors._id: 674d75a01e1ca74a41c2123e
paper_authors.name: Micol Altomare
paper_authors._id: 674d75a01e1ca74a41c2123f
paper_authors.name: Mohamed A. Haggag
paper_authors._id: 674d75a01e1ca74a41c21240
paper_authors.name: Snegha A
paper_authors._id: 674d75a01e1ca74a41c21241
paper_authors.name: Alfonso Amayuelas
paper_authors._id: 674d75a01e1ca74a41c21242
paper_authors.name: Azril Hafizi Amirudin
paper_authors._id: 674d75a01e1ca74a41c21243
paper_authors.name: Viraat Aryabumi
paper_authors._id: 674d75a01e1ca74a41c21244
paper_authors.name: Danylo Boiko
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-03T08:16:29.785Z
paper_authors._id: 6655a52cbb6e96974bcbe289
paper_authors.avatarUrl: /avatars/7175adff262cb38bce42a10ddee354c3.svg
paper_authors.fullname: Danylo Boiko
paper_authors.type: user
paper_authors.user: danylo-boiko
paper_authors._id: 674d75a01e1ca74a41c21245
paper_authors.name: Michael Chang
paper_authors._id: 674d75a01e1ca74a41c21246
paper_authors.name: Jenny Chim
paper_authors._id: 674d75a01e1ca74a41c21247
paper_authors.name: Gal Cohen
paper_authors._id: 674d75a01e1ca74a41c21248
paper_authors.name: Aditya Kumar Dalmia
paper_authors._id: 674d75a01e1ca74a41c21249
paper_authors.name: Abraham Diress
paper_authors._id: 674d75a01e1ca74a41c2124a
paper_authors.name: Sharad Duwal
paper_authors._id: 674d75a01e1ca74a41c2124b
paper_authors.name: Daniil Dzenhaliou
paper_authors._id: 674d75a01e1ca74a41c2124c
paper_authors.name: Daniel Fernando Erazo Florez
paper_authors._id: 674d75a01e1ca74a41c2124d
paper_authors.name: Fabian Farestam
paper_authors._id: 674d75a01e1ca74a41c2124e
paper_authors.name: Joseph Marvin Imperial
paper_authors._id: 674d75a01e1ca74a41c2124f
paper_authors.name: Shayekh Bin Islam
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T17:31:54.464Z
paper_authors._id: 5e4b943a37cb5b49818287b5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e4b943a37cb5b49818287b5/Sd8QOGFY-_B1z4UlFTuPf.jpeg
paper_authors.fullname: Shayekh Islam
paper_authors.type: user
paper_authors.user: shayekh
paper_authors._id: 674d75a01e1ca74a41c21250
paper_authors.name: Perttu Isotalo
paper_authors._id: 674d75a01e1ca74a41c21251
paper_authors.name: Maral Jabbarishiviari
paper_authors._id: 674d75a01e1ca74a41c21252
paper_authors.name: Börje F. Karlsson
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-11T08:24:49.499Z
paper_authors._id: 61e52be53d6dbb1da842316a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e52be53d6dbb1da842316a/gx0WGPcOCClXPymoKglc4.jpeg
paper_authors.fullname: Börje Karlsson
paper_authors.type: user
paper_authors.user: tellarin
paper_authors._id: 674d75a01e1ca74a41c21253
paper_authors.name: Eldar Khalilov
paper_authors._id: 674d75a01e1ca74a41c21254
paper_authors.name: Christopher Klamm
paper_authors._id: 674d75a01e1ca74a41c21255
paper_authors.name: Fajri Koto
paper_authors._id: 674d75a01e1ca74a41c21256
paper_authors.name: Dominik Krzemiński
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-24T10:58:18.861Z
paper_authors._id: 662cc845616128914a3c9817
paper_authors.avatarUrl: /avatars/cd186de00f28bb866abc1ab6c4465663.svg
paper_authors.fullname: DomKrz
paper_authors.type: user
paper_authors.user: dokato
paper_authors._id: 674d75a01e1ca74a41c21257
paper_authors.name: Gabriel Adriano de Melo
paper_authors._id: 674d75a01e1ca74a41c21258
paper_authors.name: Syrielle Montariol
paper_authors._id: 674d75a01e1ca74a41c21259
paper_authors.name: Yiyang Nan
paper_authors._id: 674d75a01e1ca74a41c2125a
paper_authors.name: Joel Niklaus
paper_authors._id: 674d75a01e1ca74a41c2125b
paper_authors.name: Jekaterina Novikova
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:37:20.722Z
paper_authors._id: 6136d5e0e803998e3b48f063
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6136d5e0e803998e3b48f063/Uitq0a4W9qBvkRNJ9Eiyh.jpeg
paper_authors.fullname: Jekaterina Novikova
paper_authors.type: user
paper_authors.user: Jekaterina
paper_authors._id: 674d75a01e1ca74a41c2125c
paper_authors.name: Johan Samir Obando Ceron
paper_authors._id: 674d75a01e1ca74a41c2125d
paper_authors.name: Debjit Paul
paper_authors._id: 674d75a01e1ca74a41c2125e
paper_authors.name: Esther Ploeger
paper_authors._id: 674d75a01e1ca74a41c2125f
paper_authors.name: Jebish Purbey
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T11:16:06.831Z
paper_authors._id: 66c578770a22b2f9ab575847
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66c578770a22b2f9ab575847/-zfNho1DR3yZHDazq669-.png
paper_authors.fullname: Jebish Purbey
paper_authors.type: user
paper_authors.user: jebish7
paper_authors._id: 674d75a01e1ca74a41c21260
paper_authors.name: Swati Rajwal
paper_authors._id: 674d75a01e1ca74a41c21261
paper_authors.name: Selvan Sunitha Ravi
paper_authors._id: 674d75a01e1ca74a41c21262
paper_authors.name: Sara Rydell
paper_authors._id: 674d75a01e1ca74a41c21263
paper_authors.name: Roshan Santhosh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:34:07.639Z
paper_authors._id: 65f5a22e6bd4bac5b6fa7d1d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65f5a22e6bd4bac5b6fa7d1d/1-ftEEvA66TfDa-KL1O3x.jpeg
paper_authors.fullname: Roshan Santhosh
paper_authors.type: user
paper_authors.user: roshansk23
paper_authors._id: 674d75a01e1ca74a41c21264
paper_authors.name: Drishti Sharma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:46:36.448Z
paper_authors._id: 618c1ad1c74578e0a4a4d074
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/618c1ad1c74578e0a4a4d074/8u_AkeHt4d6xtQ8hzaffU.jpeg
paper_authors.fullname: Drishti Sharma
paper_authors.type: user
paper_authors.user: DrishtiSharma
paper_authors._id: 674d75a01e1ca74a41c21265
paper_authors.name: Marjana Prifti Skenduli
paper_authors._id: 674d75a01e1ca74a41c21266
paper_authors.name: Arshia Soltani Moakhar
paper_authors._id: 674d75a01e1ca74a41c21267
paper_authors.name: Bardia Soltani Moakhar
paper_authors._id: 674d75a01e1ca74a41c21268
paper_authors.name: Ran Tamir
paper_authors._id: 674d75a01e1ca74a41c21269
paper_authors.name: Ayush Kumar Tarun
paper_authors._id: 674d75a01e1ca74a41c2126a
paper_authors.name: Azmine Toushik Wasi
paper_authors._id: 674d75a01e1ca74a41c2126b
paper_authors.name: Thenuka Ovin Weerasinghe
paper_authors._id: 674d75a01e1ca74a41c2126c
paper_authors.name: Serhan Yilmaz
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-28T12:17:04.137Z
paper_authors._id: 64b011ca9d08245fa862e482
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b011ca9d08245fa862e482/txKvPNMXNqVSUkrJgr4Yl.jpeg
paper_authors.fullname: Serhan YILMAZ
paper_authors.type: user
paper_authors.user: serhany
paper_authors._id: 674d75a01e1ca74a41c2126d
paper_authors.name: Mike Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T08:56:04.648Z
paper_authors._id: 60d33fbbd7b174177faabd4f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60d33fbbd7b174177faabd4f/pfyv_xj2B2m2N4F4sT9zJ.jpeg
paper_authors.fullname: Mike Zhang
paper_authors.type: user
paper_authors.user: jjzha
paper_authors._id: 674d75a01e1ca74a41c2126e
paper_authors.name: Imanol Schlag
paper_authors._id: 674d75a01e1ca74a41c2126f
paper_authors.name: Marzieh Fadaee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T14:40:43.732Z
paper_authors._id: 6441042d5d600fb0951a5f99
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6441042d5d600fb0951a5f99/4CbOaYcEz99BtVAQvnGTn.jpeg
paper_authors.fullname: Marzieh Fadaee
paper_authors.type: user
paper_authors.user: MarziehFadaee
paper_authors._id: 674d75a01e1ca74a41c21270
paper_authors.name: Sara Hooker
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-20T09:30:48.624Z
paper_authors._id: 63434eb76f59b79da07dbddf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63434eb76f59b79da07dbddf/BEwmVjqPNYlqmutXG0G6e.jpeg
paper_authors.fullname: Sara Hooker
paper_authors.type: user
paper_authors.user: sarahooker
paper_authors._id: 674d75a01e1ca74a41c21271
paper_authors.name: Antoine Bosselut
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-12-02T08:57:36.004Z
paper_authors._id: 654fb1c67490049d621516fb
paper_authors.avatarUrl: /avatars/0f88177da236b1bce4a4a58fd0246884.svg
paper_authors.fullname: Antoine Bosselut
paper_authors.type: user
paper_authors.user: atcbosselut
paper_title: INCLUDE: Evaluating Multilingual Language Understanding with Regional
    Knowledge
paper_summary: The performance differential of large language models (LLM) between languages
  hinders their effective deployment in many regions, inhibiting the potential
  economic and societal value of generative AI tools in many communities.
  However, the development of functional LLMs in many languages (\ie,
  multilingual LLMs) is bottlenecked by the lack of high-quality evaluation
  resources in languages other than English. Moreover, current practices in
  multilingual benchmark construction often translate English resources, ignoring
  the regional and cultural knowledge of the environments in which multilingual
  systems would be used. In this work, we construct an evaluation suite of
  197,243 QA pairs from local exam sources to measure the capabilities of
  multilingual LLMs in a variety of regional contexts. Our novel resource,
  INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across
  44 written languages that evaluates multilingual LLMs for performance in the
  actual language environments where they would be deployed.
paper_ai_keywords: large language models (LLM) multilingual LLMs high-quality natural resources multilingual benchmark construction QA pairs INCLUDE comprehensive knowledge- and reasoning-centric benchmark multilingual systems