title: MUG-Eval: A Proxy Evaluation Framework for Multilingual Generation
    Capabilities in Any Language
summary: Evaluating text generation capabilities of large language models (LLMs) is
  challenging, particularly for low-resource languages where methods for direct
  assessment are scarce. We propose MUG-Eval, a novel framework that evaluates
  LLMs' multilingual generation capabilities by transforming existing benchmarks
  into conversational tasks and measuring the LLMs' accuracies on those tasks. We
  specifically designed these conversational tasks to require effective
  communication in the target language. Then, we simply use task success rate as
  a proxy of successful conversation generation. Our approach offers two key
  advantages: it is independent of language-specific NLP tools or annotated
  datasets, which are limited for most languages, and it does not rely on
  LLMs-as-judges, whose evaluation quality degrades outside a few high-resource
  languages. We evaluate 8 LLMs across 30 languages spanning high, mid, and
  low-resource categories, and we find that MUG-Eval correlates strongly with
  established benchmarks (r > 0.75) while enabling standardized comparisons
  across languages and models. Our framework provides a robust and
  resource-efficient solution for evaluating multilingual generation that can be
  extended to thousands of languages.
paper_authors._id: 682db49f167398cff979ec27
paper_authors.name: Seyoung Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T12:27:01.867Z
paper_authors._id: 654f3cca8cc59d5b490b805b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/654f3cca8cc59d5b490b805b/WAZZjN4q8VGp3VS2lAWyy.png
paper_authors.fullname: Seyoung Song
paper_authors.type: user
paper_authors.user: seyoungsong
paper_authors._id: 682db49f167398cff979ec28
paper_authors.name: Seogyeong Jeong
paper_authors._id: 682db49f167398cff979ec29
paper_authors.name: Eunsu Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T07:30:12.427Z
paper_authors._id: 6576ace7769f3ee9bd7b1b88
paper_authors.avatarUrl: /avatars/5b5921e54413a37afde6ce017809c86e.svg
paper_authors.fullname: Eunsu Kim
paper_authors.type: user
paper_authors.user: EunsuKim
paper_authors._id: 682db49f167398cff979ec2a
paper_authors.name: Jiho Jin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:22:47.397Z
paper_authors._id: 65e52c1c57459fa149c7fd48
paper_authors.avatarUrl: /avatars/594749486825ea3cce7d0aa2c673d644.svg
paper_authors.fullname: Jiho Jin
paper_authors.type: user
paper_authors.user: jinjh0123
paper_authors._id: 682db49f167398cff979ec2b
paper_authors.name: Dongkwan Kim
paper_authors._id: 682db49f167398cff979ec2c
paper_authors.name: Jay Shin
paper_authors._id: 682db49f167398cff979ec2d
paper_authors.name: Alice Oh
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-21T19:06:44.285Z
paper_authors._id: 60e0251ea9b5d8282481f2b7
paper_authors.avatarUrl: /avatars/43441373af054a6184c22097bfeb97e4.svg
paper_authors.fullname: Alice Oh
paper_authors.type: user
paper_authors.user: aliceoh
paper_title: MUG-Eval: A Proxy Evaluation Framework for Multilingual Generation
    Capabilities in Any Language
paper_summary: Evaluating text generation capabilities of large language models (LLMs) is
  challenging, particularly for low-resource languages where methods for direct
  assessment are scarce. We propose MUG-Eval, a novel framework that evaluates
  LLMs' multilingual generation capabilities by transforming existing benchmarks
  into conversational tasks and measuring the LLMs' accuracies on those tasks. We
  specifically designed these conversational tasks to require effective
  communication in the target language. Then, we simply use task success rate as
  a proxy of successful conversation generation. Our approach offers two key
  advantages: it is independent of language-specific NLP tools or annotated
  datasets, which are limited for most languages, and it does not rely on
  LLMs-as-judges, whose evaluation quality degrades outside a few high-resource
  languages. We evaluate 8 LLMs across 30 languages spanning high, mid, and
  low-resource categories, and we find that MUG-Eval correlates strongly with
  established benchmarks (r > 0.75) while enabling standardized comparisons
  across languages and models. Our framework provides a robust and
  resource-efficient solution for evaluating multilingual generation that can be
  extended to thousands of languages.
paper_ai_summary: MUG-Eval assesses LLMs' multilingual generation by transforming benchmarks into conversational tasks, offering a language-independent and NLP tool-free method that correlates well with established benchmarks.
paper_ai_keywords: large language models LLMs multilingual generation conversational tasks task success rate low-resource languages high-resource languages NLP tools annotated datasets MUG-Eval standardized comparisons