title: QE4PE: Word-level Quality Estimation for Human Post-Editing
summary: Word-level quality estimation (QE) detects erroneous spans in machine
  translations, which can direct and facilitate human post-editing. While the
  accuracy of word-level QE systems has been assessed extensively, their
  usability and downstream influence on the speed, quality and editing choices of
  human post-editing remain understudied. Our QE4PE study investigates the impact
  of word-level QE on machine translation (MT) post-editing in a realistic
  setting involving 42 professional post-editors across two translation
  directions. We compare four error-span highlight modalities, including
  supervised and uncertainty-based word-level QE methods, for identifying
  potential errors in the outputs of a state-of-the-art neural MT model.
  Post-editing effort and productivity are estimated by behavioral logs, while
  quality improvements are assessed by word- and segment-level human annotation.
  We find that domain, language and editors' speed are critical factors in
  determining highlights' effectiveness, with modest differences between
  human-made and automated QE highlights underlining a gap between accuracy and
  usability in professional workflows.
paper_authors._id: 67c94e6ad325e95d82f23433
paper_authors.name: Gabriele Sarti
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-06T09:24:30.601Z
paper_authors._id: 5e7749883d77a72421292d07
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e7749883d77a72421292d07/M4AmBReZk_otxCIG3o0bL.jpeg
paper_authors.fullname: Gabriele Sarti
paper_authors.type: user
paper_authors.user: gsarti
paper_authors._id: 67c94e6ad325e95d82f23434
paper_authors.name: Vilém Zouhar
paper_authors._id: 67c94e6ad325e95d82f23435
paper_authors.name: Grzegorz Chrupała
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-11T13:42:39.952Z
paper_authors._id: 62ab405c0a9f695086a247d4
paper_authors.avatarUrl: /avatars/cc2828bfcbed9bc0161263dd7e539531.svg
paper_authors.fullname: Grzegorz Chrupała
paper_authors.type: user
paper_authors.user: gchrupala
paper_authors._id: 67c94e6ad325e95d82f23436
paper_authors.name: Ana Guerberof-Arenas
paper_authors._id: 67c94e6ad325e95d82f23437
paper_authors.name: Malvina Nissim
paper_authors._id: 67c94e6ad325e95d82f23438
paper_authors.name: Arianna Bisazza
paper_title: QE4PE: Word-level Quality Estimation for Human Post-Editing
paper_summary: Word-level quality estimation (QE) detects erroneous spans in machine
  translations, which can direct and facilitate human post-editing. While the
  accuracy of word-level QE systems has been assessed extensively, their
  usability and downstream influence on the speed, quality and editing choices of
  human post-editing remain understudied. Our QE4PE study investigates the impact
  of word-level QE on machine translation (MT) post-editing in a realistic
  setting involving 42 professional post-editors across two translation
  directions. We compare four error-span highlight modalities, including
  supervised and uncertainty-based word-level QE methods, for identifying
  potential errors in the outputs of a state-of-the-art neural MT model.
  Post-editing effort and productivity are estimated by behavioral logs, while
  quality improvements are assessed by word- and segment-level human annotation.
  We find that domain, language and editors' speed are critical factors in
  determining highlights' effectiveness, with modest differences between
  human-made and automated QE highlights underlining a gap between accuracy and
  usability in professional workflows.
paper_ai_summary: The study examines the impact of word-level quality estimation on human post-editing of machine translation outputs, comparing different error-span highlighting methods and identifying critical factors affecting effectiveness.
paper_ai_keywords: word-level quality estimation machine translation error-span highlight neural MT model