title: Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs
summary: Evaluating machine-generated text remains a significant challenge in NLP,
  especially for non-English languages. Current methodologies, including
  automated metrics, human assessments, and LLM-based evaluations, predominantly
  focus on English, revealing a significant gap in multilingual evaluation
  frameworks. We introduce the Cross Lingual Auto Evaluation (CIA) Suite, an
  extensible framework that includes evaluator LLMs (Hercule) and a novel test
  set (Recon) specifically designed for multilingual evaluation. Our test set
  features 500 human-annotated instructions spanning various task capabilities
  along with human judgment scores across six languages. This would enable
  benchmarking of general-purpose multilingual LLMs and facilitate
  meta-evaluation of Evaluator LLMs. The proposed model, Hercule, is a
  cross-lingual evaluation model that addresses the scarcity of reference answers
  in the target language by learning to assign scores to responses based on
  easily available reference answers in English. Our experiments demonstrate that
  Hercule aligns more closely with human judgments compared to proprietary
  models, demonstrating the effectiveness of such cross-lingual evaluation in low
  resource scenarios. Further, it is also effective in zero-shot evaluation on
  unseen languages. This study is the first comprehensive examination of
  cross-lingual evaluation using LLMs, presenting a scalable and effective
  approach for multilingual assessment. All code, datasets, and models will be
  publicly available to enable further research in this important area.
paper_authors._id: 67121b74e82a3853eda3b069
paper_authors.name: Sumanth Doddapaneni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T09:48:58.988Z
paper_authors._id: 5fbfc7a6e366524fe8e97ccf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1673518581924-5fbfc7a6e366524fe8e97ccf.png
paper_authors.fullname: Sumanth Doddapaneni
paper_authors.type: user
paper_authors.user: sumanthd
paper_authors._id: 67121b74e82a3853eda3b06a
paper_authors.name: Mohammed Safi Ur Rahman Khan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T12:15:16.098Z
paper_authors._id: 63ef3cd11e695b35aa48bebc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/BnOgZiv1IHKbtTaFTWpAZ.png
paper_authors.fullname: Mohammed Safi Ur Rahman Khan
paper_authors.type: user
paper_authors.user: safikhan
paper_authors._id: 67121b74e82a3853eda3b06b
paper_authors.name: Dilip Venkatesh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T09:50:07.904Z
paper_authors._id: 6408278e8dca6cec91cb7e55
paper_authors.avatarUrl: /avatars/fc599425785202727b1a21b89af552bb.svg
paper_authors.fullname: Dilip Venkatesh
paper_authors.type: user
paper_authors.user: dipsivenkatesh
paper_authors._id: 67121b74e82a3853eda3b06c
paper_authors.name: Raj Dabre
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T09:50:02.568Z
paper_authors._id: 61cd4af60996e2a209ff7b96
paper_authors.avatarUrl: /avatars/7b47bfaba15dfaf37ce38cb0216f3102.svg
paper_authors.fullname: Raj Dabre
paper_authors.type: user
paper_authors.user: prajdabre
paper_authors._id: 67121b74e82a3853eda3b06d
paper_authors.name: Anoop Kunchukuttan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T09:49:57.100Z
paper_authors._id: 5fce179fb3dbf216ad31837b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1659894079257-5fce179fb3dbf216ad31837b.jpeg
paper_authors.fullname: Anoop Kunchukuttan
paper_authors.type: user
paper_authors.user: anoopk
paper_authors._id: 67121b74e82a3853eda3b06e
paper_authors.name: Mitesh M. Khapra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T09:49:50.912Z
paper_authors._id: 6551c181c0e106160e9d3297
paper_authors.avatarUrl: /avatars/d2c498d7f361f4133d1104fde72c24b9.svg
paper_authors.fullname: Mitesh Khapra
paper_authors.type: user
paper_authors.user: miteshk
paper_title: Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs
paper_summary: Evaluating machine-generated text remains a significant challenge in NLP,
  especially for non-English languages. Current methodologies, including
  automated metrics, human assessments, and LLM-based evaluations, predominantly
  focus on English, revealing a significant gap in multilingual evaluation
  frameworks. We introduce the Cross Lingual Auto Evaluation (CIA) Suite, an
  extensible framework that includes evaluator LLMs (Hercule) and a novel test
  set (Recon) specifically designed for multilingual evaluation. Our test set
  features 500 human-annotated instructions spanning various task capabilities
  along with human judgment scores across six languages. This would enable
  benchmarking of general-purpose multilingual LLMs and facilitate
  meta-evaluation of Evaluator LLMs. The proposed model, Hercule, is a
  cross-lingual evaluation model that addresses the scarcity of reference answers
  in the target language by learning to assign scores to responses based on
  easily available reference answers in English. Our experiments demonstrate that
  Hercule aligns more closely with human judgments compared to proprietary
  models, demonstrating the effectiveness of such cross-lingual evaluation in low
  resource scenarios. Further, it is also effective in zero-shot evaluation on
  unseen languages. This study is the first comprehensive examination of
  cross-lingual evaluation using LLMs, presenting a scalable and effective
  approach for multilingual assessment. All code, datasets, and models will be
  publicly available to enable further research in this important area.
paper_ai_keywords: Cross Lingual Auto Evaluation (CIA) Suite evaluator LLMs Hercule Recon multilingual evaluation frameworks LLM-based evaluations human-annotated instructions human judgment scores multilingual evaluation cross-lingual evaluation model reference answers zero-shot evaluation cross-lingual evaluation