Unsupervised Word-level Quality Estimation for Machine Translation
  Through the Lens of Annotators (Dis)agreement
Word-level quality estimation (WQE) aims to automatically identify
fine-grained error spans in machine-translated outputs and has found many uses,
including assisting translators during post-editing. Modern WQE techniques are
often expensive, involving prompting of large language models or ad-hoc
training on large amounts of human-labeled data. In this work, we investigate
efficient alternatives exploiting recent advances in language model
interpretability and uncertainty quantification to identify translation errors
from the inner workings of translation models. In our evaluation spanning 14
metrics across 12 translation directions, we quantify the impact of human label
variation on metric performance by using multiple sets of human labels. Our
results highlight the untapped potential of unsupervised metrics, the
shortcomings of supervised methods when faced with label uncertainty, and the
brittleness of single-annotator evaluation practices.
683962e4cba8ce4f5ebced9c
Gabriele Sarti
extracted_confirmed
2025-05-30T07:54:16.127Z
5e7749883d77a72421292d07
https://cdn-avatars.huggingface.co/v1/production/uploads/5e7749883d77a72421292d07/M4AmBReZk_otxCIG3o0bL.jpeg
Gabriele Sarti
user
gsarti
683962e4cba8ce4f5ebced9d
Vilém Zouhar
683962e4cba8ce4f5ebced9e
Malvina Nissim
683962e4cba8ce4f5ebced9f
Arianna Bisazza
Unsupervised Word-level Quality Estimation for Machine Translation
  Through the Lens of Annotators (Dis)agreement
Word-level quality estimation (WQE) aims to automatically identify
fine-grained error spans in machine-translated outputs and has found many uses,
including assisting translators during post-editing. Modern WQE techniques are
often expensive, involving prompting of large language models or ad-hoc
training on large amounts of human-labeled data. In this work, we investigate
efficient alternatives exploiting recent advances in language model
interpretability and uncertainty quantification to identify translation errors
from the inner workings of translation models. In our evaluation spanning 14
metrics across 12 translation directions, we quantify the impact of human label
variation on metric performance by using multiple sets of human labels. Our
results highlight the untapped potential of unsupervised metrics, the
shortcomings of supervised methods when faced with label uncertainty, and the
brittleness of single-annotator evaluation practices.
Evaluation of word-level quality estimation techniques leverages model interpretability and uncertainty quantification to identify translation errors with a focus on the impact of label variation and the performance of supervised versus unsupervised metrics.
language model interpretability uncertainty quantification word-level quality estimation translation errors human label variation unsupervised metrics supervised methods single-annotator evaluation practices