QE4PE: Word-level Quality Estimation for Human Post-Editing
Word-level quality estimation (QE) detects erroneous spans in machine
translations, which can direct and facilitate human post-editing. While the
accuracy of word-level QE systems has been assessed extensively, their
usability and downstream influence on the speed, quality and editing choices of
human post-editing remain understudied. Our QE4PE study investigates the impact
of word-level QE on machine translation (MT) post-editing in a realistic
setting involving 42 professional post-editors across two translation
directions. We compare four error-span highlight modalities, including
supervised and uncertainty-based word-level QE methods, for identifying
potential errors in the outputs of a state-of-the-art neural MT model.
Post-editing effort and productivity are estimated by behavioral logs, while
quality improvements are assessed by word- and segment-level human annotation.
We find that domain, language and editors' speed are critical factors in
determining highlights' effectiveness, with modest differences between
human-made and automated QE highlights underlining a gap between accuracy and
usability in professional workflows.
67c94e6ad325e95d82f23433
Gabriele Sarti
claimed_verified
2025-03-06T09:24:30.601Z
5e7749883d77a72421292d07
https://cdn-avatars.huggingface.co/v1/production/uploads/5e7749883d77a72421292d07/M4AmBReZk_otxCIG3o0bL.jpeg
Gabriele Sarti
user
gsarti
67c94e6ad325e95d82f23434
Vilém Zouhar
67c94e6ad325e95d82f23435
Grzegorz Chrupała
claimed_verified
2025-03-11T13:42:39.952Z
62ab405c0a9f695086a247d4
/avatars/cc2828bfcbed9bc0161263dd7e539531.svg
Grzegorz Chrupała
user
gchrupala
67c94e6ad325e95d82f23436
Ana Guerberof-Arenas
67c94e6ad325e95d82f23437
Malvina Nissim
67c94e6ad325e95d82f23438
Arianna Bisazza
QE4PE: Word-level Quality Estimation for Human Post-Editing
Word-level quality estimation (QE) detects erroneous spans in machine
translations, which can direct and facilitate human post-editing. While the
accuracy of word-level QE systems has been assessed extensively, their
usability and downstream influence on the speed, quality and editing choices of
human post-editing remain understudied. Our QE4PE study investigates the impact
of word-level QE on machine translation (MT) post-editing in a realistic
setting involving 42 professional post-editors across two translation
directions. We compare four error-span highlight modalities, including
supervised and uncertainty-based word-level QE methods, for identifying
potential errors in the outputs of a state-of-the-art neural MT model.
Post-editing effort and productivity are estimated by behavioral logs, while
quality improvements are assessed by word- and segment-level human annotation.
We find that domain, language and editors' speed are critical factors in
determining highlights' effectiveness, with modest differences between
human-made and automated QE highlights underlining a gap between accuracy and
usability in professional workflows.
The study examines the impact of word-level quality estimation on human post-editing of machine translation outputs, comparing different error-span highlighting methods and identifying critical factors affecting effectiveness.
word-level quality estimation machine translation error-span highlight neural MT model