title: BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large
    Language Models
summary: Previous multilingual benchmarks focus primarily on simple understanding
  tasks, but for large language models(LLMs), we emphasize proficiency in
  instruction following, reasoning, long context understanding, code generation,
  and so on. However, measuring these advanced capabilities across languages is
  underexplored. To address the disparity, we introduce BenchMAX, a multi-way
  multilingual evaluation benchmark that allows for fair comparisons of these
  important abilities across languages. To maintain high quality, three distinct
  native-speaking annotators independently annotate each sample within all tasks
  after the data was machine-translated from English into 16 other languages.
  Additionally, we present a novel translation challenge stemming from dataset
  construction. Extensive experiments on BenchMAX reveal varying effectiveness of
  core capabilities across languages, highlighting performance gaps that cannot
  be bridged by simply scaling up model size. BenchMAX serves as a comprehensive
  multilingual evaluation platform, providing a promising test bed to promote the
  development of multilingual language models. The dataset and code are publicly
  accessible.
paper_authors._id: 67ac4e046b8c86e0cc7988f0
paper_authors.name: Xu Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T08:25:17.555Z
paper_authors._id: 649d1d4c379eada9a580cf59
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649d1d4c379eada9a580cf59/ucXv7KoJDEB3Phgn-Dn5E.png
paper_authors.fullname: xuhuang
paper_authors.type: user
paper_authors.user: xuhuang87
paper_authors._id: 67ac4e046b8c86e0cc7988f1
paper_authors.name: Wenhao Zhu
paper_authors._id: 67ac4e046b8c86e0cc7988f2
paper_authors.name: Hanxu Hu
paper_authors._id: 67ac4e046b8c86e0cc7988f3
paper_authors.name: Conghui He
paper_authors._id: 67ac4e046b8c86e0cc7988f4
paper_authors.name: Lei Li
paper_authors._id: 67ac4e046b8c86e0cc7988f5
paper_authors.name: Shujian Huang
paper_authors._id: 67ac4e046b8c86e0cc7988f6
paper_authors.name: Fei Yuan
paper_title: BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large
    Language Models
paper_summary: Previous multilingual benchmarks focus primarily on simple understanding
  tasks, but for large language models(LLMs), we emphasize proficiency in
  instruction following, reasoning, long context understanding, code generation,
  and so on. However, measuring these advanced capabilities across languages is
  underexplored. To address the disparity, we introduce BenchMAX, a multi-way
  multilingual evaluation benchmark that allows for fair comparisons of these
  important abilities across languages. To maintain high quality, three distinct
  native-speaking annotators independently annotate each sample within all tasks
  after the data was machine-translated from English into 16 other languages.
  Additionally, we present a novel translation challenge stemming from dataset
  construction. Extensive experiments on BenchMAX reveal varying effectiveness of
  core capabilities across languages, highlighting performance gaps that cannot
  be bridged by simply scaling up model size. BenchMAX serves as a comprehensive
  multilingual evaluation platform, providing a promising test bed to promote the
  development of multilingual language models. The dataset and code are publicly
  accessible.
paper_ai_summary: BenchMAX evaluates multilingual large language models across advanced capabilities like instruction following, reasoning, and code generation, revealing language-specific performance gaps.
paper_ai_keywords: multilingual benchmarks large language models (LLMs) instruction following reasoning long context understanding code generation BenchMAX native-speaking annotators machine translation translation challenge multilingual evaluation platform