MUG-Eval: A Proxy Evaluation Framework for Multilingual Generation
  Capabilities in Any Language
Evaluating text generation capabilities of large language models (LLMs) is
challenging, particularly for low-resource languages where methods for direct
assessment are scarce. We propose MUG-Eval, a novel framework that evaluates
LLMs' multilingual generation capabilities by transforming existing benchmarks
into conversational tasks and measuring the LLMs' accuracies on those tasks. We
specifically designed these conversational tasks to require effective
communication in the target language. Then, we simply use task success rate as
a proxy of successful conversation generation. Our approach offers two key
advantages: it is independent of language-specific NLP tools or annotated
datasets, which are limited for most languages, and it does not rely on
LLMs-as-judges, whose evaluation quality degrades outside a few high-resource
languages. We evaluate 8 LLMs across 30 languages spanning high, mid, and
low-resource categories, and we find that MUG-Eval correlates strongly with
established benchmarks (r > 0.75) while enabling standardized comparisons
across languages and models. Our framework provides a robust and
resource-efficient solution for evaluating multilingual generation that can be
extended to thousands of languages.
682db49f167398cff979ec27
Seyoung Song
claimed_verified
2025-05-21T12:27:01.867Z
654f3cca8cc59d5b490b805b
https://cdn-avatars.huggingface.co/v1/production/uploads/654f3cca8cc59d5b490b805b/WAZZjN4q8VGp3VS2lAWyy.png
Seyoung Song
user
seyoungsong
682db49f167398cff979ec28
Seogyeong Jeong
682db49f167398cff979ec29
Eunsu Kim
claimed_verified
2025-06-05T07:30:12.427Z
6576ace7769f3ee9bd7b1b88
/avatars/5b5921e54413a37afde6ce017809c86e.svg
Eunsu Kim
user
EunsuKim
682db49f167398cff979ec2a
Jiho Jin
claimed_verified
2025-05-26T08:22:47.397Z
65e52c1c57459fa149c7fd48
/avatars/594749486825ea3cce7d0aa2c673d644.svg
Jiho Jin
user
jinjh0123
682db49f167398cff979ec2b
Dongkwan Kim
682db49f167398cff979ec2c
Jay Shin
682db49f167398cff979ec2d
Alice Oh
extracted_confirmed
2025-05-21T19:06:44.285Z
60e0251ea9b5d8282481f2b7
/avatars/43441373af054a6184c22097bfeb97e4.svg
Alice Oh
user
aliceoh
MUG-Eval: A Proxy Evaluation Framework for Multilingual Generation
  Capabilities in Any Language
Evaluating text generation capabilities of large language models (LLMs) is
challenging, particularly for low-resource languages where methods for direct
assessment are scarce. We propose MUG-Eval, a novel framework that evaluates
LLMs' multilingual generation capabilities by transforming existing benchmarks
into conversational tasks and measuring the LLMs' accuracies on those tasks. We
specifically designed these conversational tasks to require effective
communication in the target language. Then, we simply use task success rate as
a proxy of successful conversation generation. Our approach offers two key
advantages: it is independent of language-specific NLP tools or annotated
datasets, which are limited for most languages, and it does not rely on
LLMs-as-judges, whose evaluation quality degrades outside a few high-resource
languages. We evaluate 8 LLMs across 30 languages spanning high, mid, and
low-resource categories, and we find that MUG-Eval correlates strongly with
established benchmarks (r > 0.75) while enabling standardized comparisons
across languages and models. Our framework provides a robust and
resource-efficient solution for evaluating multilingual generation that can be
extended to thousands of languages.
MUG-Eval assesses LLMs' multilingual generation by transforming benchmarks into conversational tasks, offering a language-independent and NLP tool-free method that correlates well with established benchmarks.
large language models LLMs multilingual generation conversational tasks task success rate low-resource languages high-resource languages NLP tools annotated datasets MUG-Eval standardized comparisons