title: GlotEval: A Test Suite for Massively Multilingual Evaluation of Large
    Language Models
summary: Large language models (LLMs) are advancing at an unprecedented pace globally,
  with regions increasingly adopting these models for applications in their
  primary language. Evaluation of these models in diverse linguistic
  environments, especially in low-resource languages, has become a major
  challenge for academia and industry. Existing evaluation frameworks are
  disproportionately focused on English and a handful of high-resource languages,
  thereby overlooking the realistic performance of LLMs in multilingual and
  lower-resource scenarios. To address this gap, we introduce GlotEval, a
  lightweight framework designed for massively multilingual evaluation.
  Supporting seven key tasks (machine translation, text classification,
  summarization, open-ended generation, reading comprehension, sequence labeling,
  and intrinsic evaluation), spanning over dozens to hundreds of languages,
  GlotEval highlights consistent multilingual benchmarking, language-specific
  prompt templates, and non-English-centric machine translation. This enables a
  precise diagnosis of model strengths and weaknesses in diverse linguistic
  contexts. A multilingual translation case study demonstrates GlotEval's
  applicability for multilingual and language-specific evaluations.
paper_authors._id: 67f50d054a7ad6225a4c6572
paper_authors.name: Hengyu Luo
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-04-08T11:48:23.178Z
paper_authors._id: 64de06e5a8b70ab6b35a7c89
paper_authors.avatarUrl: /avatars/45f287a9c07d3fe04c0782a6a99e9743.svg
paper_authors.fullname: Hengyu Luo
paper_authors.type: user
paper_authors.user: zuenmin
paper_authors._id: 67f50d054a7ad6225a4c6573
paper_authors.name: Zihao Li
paper_authors._id: 67f50d054a7ad6225a4c6574
paper_authors.name: Joseph Attieh
paper_authors._id: 67f50d054a7ad6225a4c6575
paper_authors.name: Sawal Devkota
paper_authors._id: 67f50d054a7ad6225a4c6576
paper_authors.name: Ona de Gibert
paper_authors._id: 67f50d054a7ad6225a4c6577
paper_authors.name: Shaoxiong Ji
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:51:56.626Z
paper_authors._id: 617a92e16f37340367d5d791
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/617a92e16f37340367d5d791/omgyzmaF90KBLa3YgFxhS.png
paper_authors.fullname: Shaoxiong
paper_authors.type: user
paper_authors.user: jisx
paper_authors._id: 67f50d054a7ad6225a4c6578
paper_authors.name: Peiqin Lin
paper_authors._id: 67f50d054a7ad6225a4c6579
paper_authors.name: Bhavani Sai Praneeth Varma Mantina
paper_authors._id: 67f50d054a7ad6225a4c657a
paper_authors.name: Ananda Sreenidhi
paper_authors._id: 67f50d054a7ad6225a4c657b
paper_authors.name: Raúl Vázquez
paper_authors._id: 67f50d054a7ad6225a4c657c
paper_authors.name: Mengjie Wang
paper_authors._id: 67f50d054a7ad6225a4c657d
paper_authors.name: Samea Yusofi
paper_authors._id: 67f50d054a7ad6225a4c657e
paper_authors.name: Jörg Tiedemann
paper_title: GlotEval: A Test Suite for Massively Multilingual Evaluation of Large
    Language Models
paper_summary: Large language models (LLMs) are advancing at an unprecedented pace globally,
  with regions increasingly adopting these models for applications in their
  primary language. Evaluation of these models in diverse linguistic
  environments, especially in low-resource languages, has become a major
  challenge for academia and industry. Existing evaluation frameworks are
  disproportionately focused on English and a handful of high-resource languages,
  thereby overlooking the realistic performance of LLMs in multilingual and
  lower-resource scenarios. To address this gap, we introduce GlotEval, a
  lightweight framework designed for massively multilingual evaluation.
  Supporting seven key tasks (machine translation, text classification,
  summarization, open-ended generation, reading comprehension, sequence labeling,
  and intrinsic evaluation), spanning over dozens to hundreds of languages,
  GlotEval highlights consistent multilingual benchmarking, language-specific
  prompt templates, and non-English-centric machine translation. This enables a
  precise diagnosis of model strengths and weaknesses in diverse linguistic
  contexts. A multilingual translation case study demonstrates GlotEval's
  applicability for multilingual and language-specific evaluations.
paper_ai_summary: GlotEval is a lightweight framework that enables consistent multilingual benchmarking across seven key tasks, addressing the evaluation gap of large language models in low-resource languages.
paper_ai_keywords: large language models multilingual evaluation machine translation text classification summarization open-ended generation reading comprehension sequence labeling intrinsic evaluation language-specific prompt templates non-English-centric machine translation