Cross-lingual Human-Preference Alignment for Neural Machine Translation with Direct Quality Optimization
<|reference_start|>Cross-lingual Human-Preference Alignment for Neural Machine Translation with Direct Quality Optimization: Reinforcement Learning from Human Feedback (RLHF) and derivative techniques like Direct Preference Optimization (DPO) are task-alignment algorithms used to repurpose general, foundational models for specific tasks. We show that applying task-alignment to neural machine translation (NMT) addresses an existing task--data mismatch in NMT, leading to improvements across all languages of a multilingual model, even when task-alignment is only applied to a subset of those languages. We do so by introducing Direct Quality Optimization (DQO), a variant of DPO leveraging a pre-trained translation quality estimation model as a proxy for human preferences, and verify the improvements with both automatic metrics and human evaluation.<|reference_end|>
@article{uhlig2024cross-lingual,
    title={Cross-lingual Human-Preference Alignment for Neural Machine Translation
  with Direct Quality Optimization},
    author={Kaden Uhlig, Joern Wuebker, Raphael Reinauer, John DeNero},
    journal={arXiv preprint arXiv:2409.17673},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.17673},
    primaryClass={cs.CL}
}