GlotEval: A Test Suite for Massively Multilingual Evaluation of Large
  Language Models
Large language models (LLMs) are advancing at an unprecedented pace globally,
with regions increasingly adopting these models for applications in their
primary language. Evaluation of these models in diverse linguistic
environments, especially in low-resource languages, has become a major
challenge for academia and industry. Existing evaluation frameworks are
disproportionately focused on English and a handful of high-resource languages,
thereby overlooking the realistic performance of LLMs in multilingual and
lower-resource scenarios. To address this gap, we introduce GlotEval, a
lightweight framework designed for massively multilingual evaluation.
Supporting seven key tasks (machine translation, text classification,
summarization, open-ended generation, reading comprehension, sequence labeling,
and intrinsic evaluation), spanning over dozens to hundreds of languages,
GlotEval highlights consistent multilingual benchmarking, language-specific
prompt templates, and non-English-centric machine translation. This enables a
precise diagnosis of model strengths and weaknesses in diverse linguistic
contexts. A multilingual translation case study demonstrates GlotEval's
applicability for multilingual and language-specific evaluations.
67f50d054a7ad6225a4c6572
Hengyu Luo
extracted_pending
2025-04-08T11:48:23.178Z
64de06e5a8b70ab6b35a7c89
/avatars/45f287a9c07d3fe04c0782a6a99e9743.svg
Hengyu Luo
user
hengyu-luo
67f50d054a7ad6225a4c6573
Zihao Li
67f50d054a7ad6225a4c6574
Joseph Attieh
67f50d054a7ad6225a4c6575
Sawal Devkota
67f50d054a7ad6225a4c6576
Ona de Gibert
67f50d054a7ad6225a4c6577
Shaoxiong Ji
claimed_verified
2025-06-03T08:51:56.626Z
617a92e16f37340367d5d791
https://cdn-avatars.huggingface.co/v1/production/uploads/617a92e16f37340367d5d791/omgyzmaF90KBLa3YgFxhS.png
Shaoxiong
user
jisx
67f50d054a7ad6225a4c6578
Peiqin Lin
67f50d054a7ad6225a4c6579
Bhavani Sai Praneeth Varma Mantina
67f50d054a7ad6225a4c657a
Ananda Sreenidhi
67f50d054a7ad6225a4c657b
Raúl Vázquez
67f50d054a7ad6225a4c657c
Mengjie Wang
67f50d054a7ad6225a4c657d
Samea Yusofi
67f50d054a7ad6225a4c657e
Jörg Tiedemann
GlotEval: A Test Suite for Massively Multilingual Evaluation of Large
  Language Models
Large language models (LLMs) are advancing at an unprecedented pace globally,
with regions increasingly adopting these models for applications in their
primary language. Evaluation of these models in diverse linguistic
environments, especially in low-resource languages, has become a major
challenge for academia and industry. Existing evaluation frameworks are
disproportionately focused on English and a handful of high-resource languages,
thereby overlooking the realistic performance of LLMs in multilingual and
lower-resource scenarios. To address this gap, we introduce GlotEval, a
lightweight framework designed for massively multilingual evaluation.
Supporting seven key tasks (machine translation, text classification,
summarization, open-ended generation, reading comprehension, sequence labeling,
and intrinsic evaluation), spanning over dozens to hundreds of languages,
GlotEval highlights consistent multilingual benchmarking, language-specific
prompt templates, and non-English-centric machine translation. This enables a
precise diagnosis of model strengths and weaknesses in diverse linguistic
contexts. A multilingual translation case study demonstrates GlotEval's
applicability for multilingual and language-specific evaluations.
GlotEval is a lightweight framework that enables consistent multilingual benchmarking across seven key tasks, addressing the evaluation gap of large language models in low-resource languages.
large language models multilingual evaluation machine translation text classification summarization open-ended generation reading comprehension sequence labeling intrinsic evaluation language-specific prompt templates non-English-centric machine translation