title: Are Large Language Model-based Evaluators the Solution to Scaling Up
    Multilingual Evaluation?
summary: Large Language Models (LLMs) have demonstrated impressive performance on
  Natural Language Processing (NLP) tasks, such as Question Answering,
  Summarization, and Classification. The use of LLMs as evaluators, that can rank
  or score the output of other models (usually LLMs) has become increasingly
  popular, due to the limitations of current evaluation techniques including the
  lack of appropriate benchmarks, metrics, cost, and access to human annotators.
  While LLMs are capable of handling approximately 100 languages, the majority of
  languages beyond the top 20 lack systematic evaluation across various tasks,
  metrics, and benchmarks. This creates an urgent need to scale up multilingual
  evaluation to ensure a precise understanding of LLM performance across diverse
  languages. LLM-based evaluators seem like the perfect solution to this problem,
  as they do not require human annotators, human-created references, or
  benchmarks and can theoretically be used to evaluate any language covered by
  the LLM. In this paper, we investigate whether LLM-based evaluators can help
  scale up multilingual evaluation. Specifically, we calibrate LLM-based
  evaluation against 20k human judgments of five metrics across three
  text-generation tasks in eight languages. Our findings indicate that LLM-based
  evaluators may exhibit bias towards higher scores and should be used with
  caution and should always be calibrated with a dataset of native speaker
  judgments, particularly in low-resource and non-Latin script languages.
paper_authors._id: 6503bf26d9c245846b2f6a37
paper_authors.name: Rishav Hada
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-15T09:24:00.888Z
paper_authors._id: 605229ff72705624f02496e2
paper_authors.avatarUrl: /avatars/3a3353c8a509ddfc1096d1eb6d5ab576.svg
paper_authors.fullname: Rishav Hada
paper_authors.type: user
paper_authors.user: hadarishav
paper_authors._id: 6503bf26d9c245846b2f6a38
paper_authors.name: Varun Gumma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-15T09:24:15.868Z
paper_authors._id: 64e5f244abf028d943a38350
paper_authors.avatarUrl: /avatars/52ac934a28e354f5962e8a8b02ec887a.svg
paper_authors.fullname: Varun Gumma
paper_authors.type: user
paper_authors.user: VarunGumma
paper_authors._id: 6503bf26d9c245846b2f6a39
paper_authors.name: Adrian de Wynter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-15T09:24:29.206Z
paper_authors._id: 61fb144d7884fbb4a3c5983c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1646244925841-61fb144d7884fbb4a3c5983c.jpeg
paper_authors.fullname: Adrian de Wynter
paper_authors.type: user
paper_authors.user: adewynter
paper_authors._id: 6503bf26d9c245846b2f6a3a
paper_authors.name: Harshita Diddee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-15T09:24:37.475Z
paper_authors._id: 610e4e8370e452eb73ffbc87
paper_authors.avatarUrl: /avatars/ea82b568f479f7df175da7f412383952.svg
paper_authors.fullname: Diddee
paper_authors.type: user
paper_authors.user: HarshitaDiddee
paper_authors._id: 6503bf26d9c245846b2f6a3b
paper_authors.name: Mohamed Ahmed
paper_authors._id: 6503bf26d9c245846b2f6a3c
paper_authors.name: Monojit Choudhury
paper_authors._id: 6503bf26d9c245846b2f6a3d
paper_authors.name: Kalika Bali
paper_authors._id: 6503bf26d9c245846b2f6a3e
paper_authors.name: Sunayana Sitaram
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-11-14T03:56:37.829Z
paper_authors._id: 63edb3b7684767daecb04417
paper_authors.avatarUrl: /avatars/9265619db1a1bd3c2470ef22e0d95dc4.svg
paper_authors.fullname: Sunayana Sitaram
paper_authors.type: user
paper_authors.user: Sunayana
paper_title: Are Large Language Model-based Evaluators the Solution to Scaling Up
    Multilingual Evaluation?
paper_summary: Large Language Models (LLMs) have demonstrated impressive performance on
  Natural Language Processing (NLP) tasks, such as Question Answering,
  Summarization, and Classification. The use of LLMs as evaluators, that can rank
  or score the output of other models (usually LLMs) has become increasingly
  popular, due to the limitations of current evaluation techniques including the
  lack of appropriate benchmarks, metrics, cost, and access to human annotators.
  While LLMs are capable of handling approximately 100 languages, the majority of
  languages beyond the top 20 lack systematic evaluation across various tasks,
  metrics, and benchmarks. This creates an urgent need to scale up multilingual
  evaluation to ensure a precise understanding of LLM performance across diverse
  languages. LLM-based evaluators seem like the perfect solution to this problem,
  as they do not require human annotators, human-created references, or
  benchmarks and can theoretically be used to evaluate any language covered by
  the LLM. In this paper, we investigate whether LLM-based evaluators can help
  scale up multilingual evaluation. Specifically, we calibrate LLM-based
  evaluation against 20k human judgments of five metrics across three
  text-generation tasks in eight languages. Our findings indicate that LLM-based
  evaluators may exhibit bias towards higher scores and should be used with
  caution and should always be calibrated with a dataset of native speaker
  judgments, particularly in low-resource and non-Latin script languages.
paper_ai_summary: LLM-based evaluators show promise for scaling multilingual NLP tasks evaluation but may have biases toward higher scores and require calibration with native speaker judgments.
paper_ai_keywords: Large Language Models LLMs Natural Language Processing NLP Question Answering Summarization Classification multilingual evaluation human judgments text-generation tasks non-Latin script languages