title: MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual
    Alignment
summary: English-centric large language models (LLMs) often show strong multilingual
  capabilities. However, the multilingual performance of these models remains
  unclear and is not thoroughly evaluated for many languages. Most benchmarks for
  multilinguality focus on classic NLP tasks, or cover a minimal number of
  languages. We introduce MEXA, a method for assessing the multilingual
  capabilities of pre-trained English-centric LLMs using parallel sentences,
  which are available for more languages than existing downstream tasks. MEXA
  leverages the fact that English-centric LLMs use English as a kind of pivot
  language in their intermediate layers. It computes the alignment between
  English and non-English languages using parallel sentences to evaluate the
  transfer of language understanding from English to other languages. This
  alignment can be used to estimate model performance in other languages. We
  conduct studies using various parallel datasets (FLORES-200 and Bible), models
  (Llama family, Gemma family, Mistral, and OLMo), and established downstream
  tasks (Belebele, m-MMLU, and m-ARC). We explore different methods to compute
  embeddings in decoder-only models. Our results show that MEXA, in its default
  settings, achieves a statistically significant average Pearson correlation of
  0.90 with three established downstream tasks across nine models and two
  parallel datasets. This suggests that MEXA is a reliable method for estimating
  the multilingual capabilities of English-centric LLMs, providing a clearer
  understanding of their multilingual potential and the inner workings of LLMs.
  Leaderboard: https://huggingface.co/spaces/cis-lmu/Mexa, Code:
  https://github.com/cisnlp/Mexa.
paper_authors._id: 6707e251500b1d1f3b493f2b
paper_authors.name: Amir Hossein Kargaran
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:31:22.705Z
paper_authors._id: 61bf84c8ca59d6d196a1b4e8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61bf84c8ca59d6d196a1b4e8/L_NvUwlMYcye9X35z6f7e.jpeg
paper_authors.fullname: Amir Hossein Kargaran
paper_authors.type: user
paper_authors.user: kargaranamir
paper_authors._id: 6707e251500b1d1f3b493f2c
paper_authors.name: Ali Modarressi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T14:19:35.792Z
paper_authors._id: 60e4738a8c0ddd18fc27ff88
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e4738a8c0ddd18fc27ff88/lpLeeIW8r85RTY4fGZTva.jpeg
paper_authors.fullname: Ali Modarressi
paper_authors.type: user
paper_authors.user: amodaresi
paper_authors._id: 6707e251500b1d1f3b493f2d
paper_authors.name: Nafiseh Nikeghbal
paper_authors._id: 6707e251500b1d1f3b493f2e
paper_authors.name: Jana Diesner
paper_authors._id: 6707e251500b1d1f3b493f2f
paper_authors.name: François Yvon
paper_authors._id: 6707e251500b1d1f3b493f30
paper_authors.name: Hinrich Schütze
paper_title: MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual
    Alignment
paper_summary: English-centric large language models (LLMs) often show strong multilingual
  capabilities. However, the multilingual performance of these models remains
  unclear and is not thoroughly evaluated for many languages. Most benchmarks for
  multilinguality focus on classic NLP tasks, or cover a minimal number of
  languages. We introduce MEXA, a method for assessing the multilingual
  capabilities of pre-trained English-centric LLMs using parallel sentences,
  which are available for more languages than existing downstream tasks. MEXA
  leverages the fact that English-centric LLMs use English as a kind of pivot
  language in their intermediate layers. It computes the alignment between
  English and non-English languages using parallel sentences to evaluate the
  transfer of language understanding from English to other languages. This
  alignment can be used to estimate model performance in other languages. We
  conduct studies using various parallel datasets (FLORES-200 and Bible), models
  (Llama family, Gemma family, Mistral, and OLMo), and established downstream
  tasks (Belebele, m-MMLU, and m-ARC). We explore different methods to compute
  embeddings in decoder-only models. Our results show that MEXA, in its default
  settings, achieves a statistically significant average Pearson correlation of
  0.90 with three established downstream tasks across nine models and two
  parallel datasets. This suggests that MEXA is a reliable method for estimating
  the multilingual capabilities of English-centric LLMs, providing a clearer
  understanding of their multilingual potential and the inner workings of LLMs.
  Leaderboard: https://huggingface.co/spaces/cis-lmu/Mexa, Code:
  https://github.com/cisnlp/Mexa.
paper_ai_summary: MEXA assesses the multilingual capabilities of English-centric LLMs using parallel sentences, providing a reliable method for estimating their performance across various languages and tasks.
paper_ai_keywords: pre-trained English-centric LLMs parallel sentences pivot language language understanding alignment embeddings decoder-only models Belebele m-MMLU m-ARC Pearson correlation