RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs
<|reference_start|>RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs: Preference optimization techniques have become a standard final stage for training state-of-art large language models (LLMs). However, despite widespread adoption, the vast majority of work to-date has focused on first-class citizen languages like English and Chinese. This captures a small fraction of the languages in the world, but also makes it unclear which aspects of current state-of-the-art research transfer to a multilingual setting. In this work, we perform an exhaustive study to achieve a new state-of-the-art in aligning multilingual LLMs. We introduce a novel, scalable method for generating high-quality multilingual feedback data to balance data coverage. We establish the benefits of cross-lingual transfer and increased dataset size in preference training. Our preference-trained model achieves a 54.4% win-rate against Aya 23 8B, the current state-of-the-art multilingual LLM in its parameter class, and a 69.5% win-rate or higher against widely used models like Gemma-1.1-7B-it, Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.3. As a result of our study, we expand the frontier of alignment techniques to 23 languages covering half of the world's population.<|reference_end|>
@article{dang2024rlhf,
    title={RLHF Can Speak Many Languages: Unlocking Multilingual Preference
  Optimization for LLMs},
    author={John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer, Ahmet
  "Ust"un, Sara Hooker},
    journal={arXiv preprint arXiv:2407.02552},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.02552},
    primaryClass={cs.CL cs.AI cs.LG}
}