title: The Devil is in the Errors: Leveraging Large Language Models for
    Fine-grained Machine Translation Evaluation
summary: Automatic evaluation of machine translation (MT) is a critical tool driving
  the rapid iterative development of MT systems. While considerable progress has
  been made on estimating a single scalar quality score, current metrics lack the
  informativeness of more detailed schemes that annotate individual errors, such
  as Multidimensional Quality Metrics (MQM). In this paper, we help fill this gap
  by proposing AutoMQM, a prompting technique which leverages the reasoning and
  in-context learning capabilities of large language models (LLMs) and asks them
  to identify and categorize errors in translations. We start by evaluating
  recent LLMs, such as PaLM and PaLM-2, through simple score prediction
  prompting, and we study the impact of labeled data through in-context learning
  and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that
  it improves performance compared to just prompting for scores (with
  particularly large gains for larger models) while providing interpretability
  through error spans that align with human annotations.
paper_authors._id: 64db1421322a5774e09c01e2
paper_authors.name: Patrick Fernandes
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-15T13:19:11.910Z
paper_authors._id: 6354206b12edd0ed5dc235f4
paper_authors.avatarUrl: /avatars/c2f5872eb4cad979635293c0af47db03.svg
paper_authors.fullname: Patrick Fernandes
paper_authors.type: user
paper_authors.user: patricksf
paper_authors._id: 64db1421322a5774e09c01e3
paper_authors.name: Daniel Deutsch
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-15T13:19:28.461Z
paper_authors._id: 64d41bd5919f6f802e738af7
paper_authors.avatarUrl: /avatars/916086b0e2850054c56a202afe46aa2f.svg
paper_authors.fullname: Daniel Deutsch
paper_authors.type: user
paper_authors.user: danieldeutsch
paper_authors._id: 64db1421322a5774e09c01e4
paper_authors.name: Mara Finkelstein
paper_authors._id: 64db1421322a5774e09c01e5
paper_authors.name: Parker Riley
paper_authors._id: 64db1421322a5774e09c01e6
paper_authors.name: André F. T. Martins
paper_authors._id: 64db1421322a5774e09c01e7
paper_authors.name: Graham Neubig
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-15T13:21:38.285Z
paper_authors._id: 60de14638bedd2315529d43f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
paper_authors.fullname: Graham Neubig
paper_authors.type: user
paper_authors.user: gneubig
paper_authors._id: 64db1421322a5774e09c01e8
paper_authors.name: Ankush Garg
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-15T13:21:57.584Z
paper_authors._id: 609bfde2138207cc2f2f3a8c
paper_authors.avatarUrl: /avatars/8bb0198645bf8aa4e473bab3b1d76b20.svg
paper_authors.fullname: garg
paper_authors.type: user
paper_authors.user: ankushgarg
paper_authors._id: 64db1421322a5774e09c01e9
paper_authors.name: Jonathan H. Clark
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-15T13:22:40.331Z
paper_authors._id: 6244a009a37b7efa7d335e47
paper_authors.avatarUrl: /avatars/465c97aa99e0377b09e2f43813da2701.svg
paper_authors.fullname: Jonathan Clark
paper_authors.type: user
paper_authors.user: jhclark
paper_authors._id: 64db1421322a5774e09c01ea
paper_authors.name: Markus Freitag
paper_authors._id: 64db1421322a5774e09c01eb
paper_authors.name: Orhan Firat
paper_title: The Devil is in the Errors: Leveraging Large Language Models for
    Fine-grained Machine Translation Evaluation
paper_summary: Automatic evaluation of machine translation (MT) is a critical tool driving
  the rapid iterative development of MT systems. While considerable progress has
  been made on estimating a single scalar quality score, current metrics lack the
  informativeness of more detailed schemes that annotate individual errors, such
  as Multidimensional Quality Metrics (MQM). In this paper, we help fill this gap
  by proposing AutoMQM, a prompting technique which leverages the reasoning and
  in-context learning capabilities of large language models (LLMs) and asks them
  to identify and categorize errors in translations. We start by evaluating
  recent LLMs, such as PaLM and PaLM-2, through simple score prediction
  prompting, and we study the impact of labeled data through in-context learning
  and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that
  it improves performance compared to just prompting for scores (with
  particularly large gains for larger models) while providing interpretability
  through error spans that align with human annotations.
paper_ai_summary: AutoMQM uses large language models to identify and categorize errors in machine translation, offering interpretability and improving performance over traditional score prediction.
paper_ai_keywords: machine translation Multidimensional Quality Metrics MQM large language models LLMs PaLM PaLM-2 score prediction in-context learning finetuning error spans human annotations