Preference Tuning For Toxicity Mitigation Generalizes Across Languages
Detoxifying multilingual Large Language Models (LLMs) has become crucial due
to their increasing global use. In this work, we explore zero-shot
cross-lingual generalization of preference tuning in detoxifying LLMs. Unlike
previous studies that show limited cross-lingual generalization for other
safety tasks, we demonstrate that Direct Preference Optimization (DPO) training
with only English data can significantly reduce toxicity in multilingual
open-ended generations. For example, the probability of mGPT-1.3B generating
toxic continuations drops from 46.8% to 3.9% across 17 different languages
after training. Our results also extend to other multilingual LLMs, such as
BLOOM, Llama3, and Aya-23. Using mechanistic interpretability tools like causal
intervention and activation analysis, we identified the dual multilinguality
property of MLP layers in LLMs, which explains the cross-lingual generalization
of DPO. Finally, we show that bilingual sentence retrieval can predict the
cross-lingual transferability of DPO preference tuning.
667a3d632d6477209ea8b46c
Xiaochen Li
claimed_verified
2024-06-25T20:03:52.127Z
65eba11c7cca5fa91c44d902
https://cdn-avatars.huggingface.co/v1/production/uploads/65eba11c7cca5fa91c44d902/0efmgs_QhixkyX21RxTYa.jpeg
Jacob Li
user
jacobli
667a3d632d6477209ea8b46d
Zheng-Xin Yong
claimed_verified
2024-06-25T07:33:04.796Z
61424bf4f0d914a5f606a823
https://cdn-avatars.huggingface.co/v1/production/uploads/61424bf4f0d914a5f606a823/0td8lR4elBaVvJUD9Pojh.png
Yong Zheng-Xin
user
yongzx
667a3d632d6477209ea8b46e
Stephen H. Bach
admin_assigned
2024-06-25T08:34:27.134Z
608849cadf398c3b285ce95b
https://cdn-avatars.huggingface.co/v1/production/uploads/608849cadf398c3b285ce95b/roABeBreO9pZRPxzVoOPM.jpeg
Stephen Bach
user
SteveBach
Preference Tuning For Toxicity Mitigation Generalizes Across Languages
Detoxifying multilingual Large Language Models (LLMs) has become crucial due
to their increasing global use. In this work, we explore zero-shot
cross-lingual generalization of preference tuning in detoxifying LLMs. Unlike
previous studies that show limited cross-lingual generalization for other
safety tasks, we demonstrate that Direct Preference Optimization (DPO) training
with only English data can significantly reduce toxicity in multilingual
open-ended generations. For example, the probability of mGPT-1.3B generating
toxic continuations drops from 46.8% to 3.9% across 17 different languages
after training. Our results also extend to other multilingual LLMs, such as
BLOOM, Llama3, and Aya-23. Using mechanistic interpretability tools like causal
intervention and activation analysis, we identified the dual multilinguality
property of MLP layers in LLMs, which explains the cross-lingual generalization
of DPO. Finally, we show that bilingual sentence retrieval can predict the
cross-lingual transferability of DPO preference tuning.
Direct Preference Optimization (DPO) trained on English data significantly reduces toxicity in multilingual Large Language Models, attributed to the dual multilinguality property of MLP layers.
Direct Preference Optimization (DPO) multilingual Large Language Models (LLMs) zero-shot cross-lingual generalization mGPT-1.3B BLOOM Llama3 Aya-23 causal intervention activation analysis MLP layers bilingual sentence retrieval