Preference Tuning For Toxicity Mitigation Generalizes Across Languages
<|reference_start|>Preference Tuning For Toxicity Mitigation Generalizes Across Languages: Detoxifying multilingual Large Language Models (LLMs) has become crucial due to their increasing global use. In this work, we explore zero-shot cross-lingual generalization of preference tuning in detoxifying LLMs. Unlike previous studies that show limited cross-lingual generalization for other safety tasks, we demonstrate that Direct Preference Optimization (DPO) training with only English data can significantly reduce toxicity in multilingual open-ended generations. For example, the probability of mGPT-1.3B generating toxic continuations drops from 46.8% to 3.9% across 17 different languages after training. Our results also extend to other multilingual LLMs, such as BLOOM, Llama3, and Aya-23. Using mechanistic interpretability tools like causal intervention and activation analysis, we identified the dual multilinguality property of MLP layers in LLMs, which explains the cross-lingual generalization of DPO. Finally, we show that bilingual sentence retrieval can predict the cross-lingual transferability of DPO preference tuning.<|reference_end|>
@article{li2024preference,
    title={Preference Tuning For Toxicity Mitigation Generalizes Across Languages},
    author={Xiaochen Li, Zheng-Xin Yong, Stephen H. Bach},
    journal={arXiv preprint arXiv:2406.16235},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.16235},
    primaryClass={cs.CL cs.AI cs.CR cs.LG}
}