The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122
  Language Variants
We present Belebele, a multiple-choice machine reading comprehension (MRC)
dataset spanning 122 language variants. Significantly expanding the language
coverage of natural language understanding (NLU) benchmarks, this dataset
enables the evaluation of text models in high-, medium-, and low-resource
languages. Each question is based on a short passage from the Flores-200
dataset and has four multiple-choice answers. The questions were carefully
curated to discriminate between models with different levels of general
language comprehension. The English dataset on its own proves difficult enough
to challenge state-of-the-art language models. Being fully parallel, this
dataset enables direct comparison of model performance across all languages. We
use this dataset to evaluate the capabilities of multilingual masked language
models (MLMs) and large language models (LLMs). We present extensive results
and find that despite significant cross-lingual transfer in English-centric
LLMs, much smaller MLMs pretrained on balanced multilingual data still
understand far more languages. We also observe that larger vocabulary size and
conscious vocabulary construction correlate with better performance on
low-resource languages. Overall, Belebele opens up new avenues for evaluating
and analyzing the multilingual capabilities of NLP systems.
64f15a1938bff02539e6aff6
Lucas Bandarkar
admin_assigned
2023-09-01T10:32:56.387Z
6435b1d9aaef013d1aeea6a0
/avatars/1f515c67f3d03f35aa6e06f93566bd21.svg
Lucas Bandarkar
user
lucasbandarkar
64f15a1938bff02539e6aff7
Davis Liang
claimed_verified
2023-09-17T19:21:13.608Z
63e15e357fbb6ae4d4efcf3a
/avatars/d134edfcea8ca2ae35ac661bbb42f777.svg
Davis Liang
user
davissblaine
64f15a1938bff02539e6aff8
Benjamin Muller
admin_assigned
2023-09-01T10:33:24.337Z
5e6b7153d4cd9779932a7600
/avatars/c1304f100758c5e2a54fd4d47f638841.svg
Benjamin Muller
user
benjamin-mlr
64f15a1938bff02539e6aff9
Mikel Artetxe
64f15a1938bff02539e6affa
Satya Narayan Shukla
64f15a1938bff02539e6affb
Donald Husa
64f15a1938bff02539e6affc
Naman Goyal
admin_assigned
2023-09-01T10:35:28.813Z
62e1cdfa3eb0730f621b5e14
https://cdn-avatars.huggingface.co/v1/production/uploads/1658965476209-noauth.jpeg
Naman Goyal
user
namangoyal
64f15a1938bff02539e6affd
Abhinandan Krishnan
64f15a1938bff02539e6affe
Luke Zettlemoyer
64f15a1938bff02539e6afff
Madian Khabsa
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122
  Language Variants
We present Belebele, a multiple-choice machine reading comprehension (MRC)
dataset spanning 122 language variants. Significantly expanding the language
coverage of natural language understanding (NLU) benchmarks, this dataset
enables the evaluation of text models in high-, medium-, and low-resource
languages. Each question is based on a short passage from the Flores-200
dataset and has four multiple-choice answers. The questions were carefully
curated to discriminate between models with different levels of general
language comprehension. The English dataset on its own proves difficult enough
to challenge state-of-the-art language models. Being fully parallel, this
dataset enables direct comparison of model performance across all languages. We
use this dataset to evaluate the capabilities of multilingual masked language
models (MLMs) and large language models (LLMs). We present extensive results
and find that despite significant cross-lingual transfer in English-centric
LLMs, much smaller MLMs pretrained on balanced multilingual data still
understand far more languages. We also observe that larger vocabulary size and
conscious vocabulary construction correlate with better performance on
low-resource languages. Overall, Belebele opens up new avenues for evaluating
and analyzing the multilingual capabilities of NLP systems.
Belebele, a multilingual MRC dataset, evaluates the performance of LLMs and MLMs across numerous language variants, highlighting the effectiveness of models trained on balanced multilingual data.
multiple-choice machine reading comprehension multilingual masked language models large language models neural language understanding low-resource languages multilingual capabilities