Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains
<|reference_start|>Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains: We introduce a new, extensive multidimensional quality metrics (MQM) annotated dataset covering 11 language pairs in the biomedical domain. We use this dataset to investigate whether machine translation (MT) metrics which are fine-tuned on human-generated MT quality judgements are robust to domain shifts between training and inference. We find that fine-tuned metrics exhibit a substantial performance drop in the unseen domain scenario relative to metrics that rely on the surface form, as well as pre-trained metrics which are not fine-tuned on MT quality judgments.<|reference_end|>
@article{zouhar2024fine-tuned,
    title={Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains},
    author={Vil'em Zouhar, Shuoyang Ding, Anna Currey, Tatyana Badeka, Jenyuan
  Wang, Brian Thompson},
    journal={arXiv preprint arXiv:2402.18747},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.18747},
    primaryClass={cs.CL cs.AI}
}