title: The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122
    Language Variants
summary: We present Belebele, a multiple-choice machine reading comprehension (MRC)
  dataset spanning 122 language variants. Significantly expanding the language
  coverage of natural language understanding (NLU) benchmarks, this dataset
  enables the evaluation of text models in high-, medium-, and low-resource
  languages. Each question is based on a short passage from the Flores-200
  dataset and has four multiple-choice answers. The questions were carefully
  curated to discriminate between models with different levels of general
  language comprehension. The English dataset on its own proves difficult enough
  to challenge state-of-the-art language models. Being fully parallel, this
  dataset enables direct comparison of model performance across all languages. We
  use this dataset to evaluate the capabilities of multilingual masked language
  models (MLMs) and large language models (LLMs). We present extensive results
  and find that despite significant cross-lingual transfer in English-centric
  LLMs, much smaller MLMs pretrained on balanced multilingual data still
  understand far more languages. We also observe that larger vocabulary size and
  conscious vocabulary construction correlate with better performance on
  low-resource languages. Overall, Belebele opens up new avenues for evaluating
  and analyzing the multilingual capabilities of NLP systems.
paper_authors._id: 64f15a1938bff02539e6aff6
paper_authors.name: Lucas Bandarkar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-01T10:32:56.387Z
paper_authors._id: 6435b1d9aaef013d1aeea6a0
paper_authors.avatarUrl: /avatars/1f515c67f3d03f35aa6e06f93566bd21.svg
paper_authors.fullname: Lucas Bandarkar
paper_authors.type: user
paper_authors.user: lucasbandarkar
paper_authors._id: 64f15a1938bff02539e6aff7
paper_authors.name: Davis Liang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-17T19:21:13.608Z
paper_authors._id: 63e15e357fbb6ae4d4efcf3a
paper_authors.avatarUrl: /avatars/d134edfcea8ca2ae35ac661bbb42f777.svg
paper_authors.fullname: Davis Liang
paper_authors.type: user
paper_authors.user: davissblaine
paper_authors._id: 64f15a1938bff02539e6aff8
paper_authors.name: Benjamin Muller
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-01T10:33:24.337Z
paper_authors._id: 5e6b7153d4cd9779932a7600
paper_authors.avatarUrl: /avatars/c1304f100758c5e2a54fd4d47f638841.svg
paper_authors.fullname: Benjamin Muller
paper_authors.type: user
paper_authors.user: benjamin-mlr
paper_authors._id: 64f15a1938bff02539e6aff9
paper_authors.name: Mikel Artetxe
paper_authors._id: 64f15a1938bff02539e6affa
paper_authors.name: Satya Narayan Shukla
paper_authors._id: 64f15a1938bff02539e6affb
paper_authors.name: Donald Husa
paper_authors._id: 64f15a1938bff02539e6affc
paper_authors.name: Naman Goyal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-01T10:35:28.813Z
paper_authors._id: 62e1cdfa3eb0730f621b5e14
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1658965476209-noauth.jpeg
paper_authors.fullname: Naman Goyal
paper_authors.type: user
paper_authors.user: namangoyal
paper_authors._id: 64f15a1938bff02539e6affd
paper_authors.name: Abhinandan Krishnan
paper_authors._id: 64f15a1938bff02539e6affe
paper_authors.name: Luke Zettlemoyer
paper_authors._id: 64f15a1938bff02539e6afff
paper_authors.name: Madian Khabsa
paper_title: The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122
    Language Variants
paper_summary: We present Belebele, a multiple-choice machine reading comprehension (MRC)
  dataset spanning 122 language variants. Significantly expanding the language
  coverage of natural language understanding (NLU) benchmarks, this dataset
  enables the evaluation of text models in high-, medium-, and low-resource
  languages. Each question is based on a short passage from the Flores-200
  dataset and has four multiple-choice answers. The questions were carefully
  curated to discriminate between models with different levels of general
  language comprehension. The English dataset on its own proves difficult enough
  to challenge state-of-the-art language models. Being fully parallel, this
  dataset enables direct comparison of model performance across all languages. We
  use this dataset to evaluate the capabilities of multilingual masked language
  models (MLMs) and large language models (LLMs). We present extensive results
  and find that despite significant cross-lingual transfer in English-centric
  LLMs, much smaller MLMs pretrained on balanced multilingual data still
  understand far more languages. We also observe that larger vocabulary size and
  conscious vocabulary construction correlate with better performance on
  low-resource languages. Overall, Belebele opens up new avenues for evaluating
  and analyzing the multilingual capabilities of NLP systems.
paper_ai_summary: Belebele, a multilingual MRC dataset, evaluates the performance of LLMs and MLMs across numerous language variants, highlighting the effectiveness of models trained on balanced multilingual data.
paper_ai_keywords: multiple-choice machine reading comprehension multilingual masked language models large language models neural language understanding low-resource languages multilingual capabilities