title: Unsupervised Word-level Quality Estimation for Machine Translation
    Through the Lens of Annotators (Dis)agreement
summary: Word-level quality estimation (WQE) aims to automatically identify
  fine-grained error spans in machine-translated outputs and has found many uses,
  including assisting translators during post-editing. Modern WQE techniques are
  often expensive, involving prompting of large language models or ad-hoc
  training on large amounts of human-labeled data. In this work, we investigate
  efficient alternatives exploiting recent advances in language model
  interpretability and uncertainty quantification to identify translation errors
  from the inner workings of translation models. In our evaluation spanning 14
  metrics across 12 translation directions, we quantify the impact of human label
  variation on metric performance by using multiple sets of human labels. Our
  results highlight the untapped potential of unsupervised metrics, the
  shortcomings of supervised methods when faced with label uncertainty, and the
  brittleness of single-annotator evaluation practices.
paper_authors._id: 683962e4cba8ce4f5ebced9c
paper_authors.name: Gabriele Sarti
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-30T07:54:16.127Z
paper_authors._id: 5e7749883d77a72421292d07
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e7749883d77a72421292d07/M4AmBReZk_otxCIG3o0bL.jpeg
paper_authors.fullname: Gabriele Sarti
paper_authors.type: user
paper_authors.user: gsarti
paper_authors._id: 683962e4cba8ce4f5ebced9d
paper_authors.name: Vilém Zouhar
paper_authors._id: 683962e4cba8ce4f5ebced9e
paper_authors.name: Malvina Nissim
paper_authors._id: 683962e4cba8ce4f5ebced9f
paper_authors.name: Arianna Bisazza
paper_title: Unsupervised Word-level Quality Estimation for Machine Translation
    Through the Lens of Annotators (Dis)agreement
paper_summary: Word-level quality estimation (WQE) aims to automatically identify
  fine-grained error spans in machine-translated outputs and has found many uses,
  including assisting translators during post-editing. Modern WQE techniques are
  often expensive, involving prompting of large language models or ad-hoc
  training on large amounts of human-labeled data. In this work, we investigate
  efficient alternatives exploiting recent advances in language model
  interpretability and uncertainty quantification to identify translation errors
  from the inner workings of translation models. In our evaluation spanning 14
  metrics across 12 translation directions, we quantify the impact of human label
  variation on metric performance by using multiple sets of human labels. Our
  results highlight the untapped potential of unsupervised metrics, the
  shortcomings of supervised methods when faced with label uncertainty, and the
  brittleness of single-annotator evaluation practices.
paper_ai_summary: Evaluation of word-level quality estimation techniques leverages model interpretability and uncertainty quantification to identify translation errors with a focus on the impact of label variation and the performance of supervised versus unsupervised metrics.
paper_ai_keywords: language model interpretability uncertainty quantification word-level quality estimation translation errors human label variation unsupervised metrics supervised methods single-annotator evaluation practices