AI-Assisted Human Evaluation of Machine Translation
<|reference_start|>AI-Assisted Human Evaluation of Machine Translation: Annually, research teams spend large amounts of money to evaluate the quality of machine translation systems (WMT, inter alia). This is expensive because it requires a lot of expert human labor. The recently adopted annotation protocol, Error Span Annotation (ESA), has annotators marking erroneous parts of the translation and then assigning a final score. A lot of the annotator time is spent on scanning the translation for possible errors. In our work, we help the annotators by pre-filling the error annotations with recall-oriented automatic quality estimation. With this AI assistance, we obtain annotations at the same quality level while cutting down the time per span annotation by half (71s/error span $\rightarrow$ 31s/error span). The biggest advantage of ESA$^\mathrm{AI}$ protocol is an accurate priming of annotators (pre-filled error spans) before they assign the final score. This also alleviates a potential automation bias, which we confirm to be low. In addition, the annotation budget can be reduced by almost 25\% with filtering of examples that the AI deems to be very likely to be correct.<|reference_end|>
@article{zouhar2024ai-assisted,
    title={AI-Assisted Human Evaluation of Machine Translation},
    author={Vil'em Zouhar, Tom Kocmi, Mrinmaya Sachan},
    journal={arXiv preprint arXiv:2406.12419},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.12419},
    primaryClass={cs.CL}
}