title: Preference Tuning For Toxicity Mitigation Generalizes Across Languages
summary: Detoxifying multilingual Large Language Models (LLMs) has become crucial due
  to their increasing global use. In this work, we explore zero-shot
  cross-lingual generalization of preference tuning in detoxifying LLMs. Unlike
  previous studies that show limited cross-lingual generalization for other
  safety tasks, we demonstrate that Direct Preference Optimization (DPO) training
  with only English data can significantly reduce toxicity in multilingual
  open-ended generations. For example, the probability of mGPT-1.3B generating
  toxic continuations drops from 46.8% to 3.9% across 17 different languages
  after training. Our results also extend to other multilingual LLMs, such as
  BLOOM, Llama3, and Aya-23. Using mechanistic interpretability tools like causal
  intervention and activation analysis, we identified the dual multilinguality
  property of MLP layers in LLMs, which explains the cross-lingual generalization
  of DPO. Finally, we show that bilingual sentence retrieval can predict the
  cross-lingual transferability of DPO preference tuning.
paper_authors._id: 667a3d632d6477209ea8b46c
paper_authors.name: Xiaochen Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T20:03:52.127Z
paper_authors._id: 65eba11c7cca5fa91c44d902
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65eba11c7cca5fa91c44d902/0efmgs_QhixkyX21RxTYa.jpeg
paper_authors.fullname: Jacob Li
paper_authors.type: user
paper_authors.user: jacobli
paper_authors._id: 667a3d632d6477209ea8b46d
paper_authors.name: Zheng-Xin Yong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T07:33:04.796Z
paper_authors._id: 61424bf4f0d914a5f606a823
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61424bf4f0d914a5f606a823/0td8lR4elBaVvJUD9Pojh.png
paper_authors.fullname: Yong Zheng-Xin
paper_authors.type: user
paper_authors.user: yongzx
paper_authors._id: 667a3d632d6477209ea8b46e
paper_authors.name: Stephen H. Bach
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:34:27.134Z
paper_authors._id: 608849cadf398c3b285ce95b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/608849cadf398c3b285ce95b/roABeBreO9pZRPxzVoOPM.jpeg
paper_authors.fullname: Stephen Bach
paper_authors.type: user
paper_authors.user: SteveBach
paper_title: Preference Tuning For Toxicity Mitigation Generalizes Across Languages
paper_summary: Detoxifying multilingual Large Language Models (LLMs) has become crucial due
  to their increasing global use. In this work, we explore zero-shot
  cross-lingual generalization of preference tuning in detoxifying LLMs. Unlike
  previous studies that show limited cross-lingual generalization for other
  safety tasks, we demonstrate that Direct Preference Optimization (DPO) training
  with only English data can significantly reduce toxicity in multilingual
  open-ended generations. For example, the probability of mGPT-1.3B generating
  toxic continuations drops from 46.8% to 3.9% across 17 different languages
  after training. Our results also extend to other multilingual LLMs, such as
  BLOOM, Llama3, and Aya-23. Using mechanistic interpretability tools like causal
  intervention and activation analysis, we identified the dual multilinguality
  property of MLP layers in LLMs, which explains the cross-lingual generalization
  of DPO. Finally, we show that bilingual sentence retrieval can predict the
  cross-lingual transferability of DPO preference tuning.
paper_ai_summary: Direct Preference Optimization (DPO) trained on English data significantly reduces toxicity in multilingual Large Language Models, attributed to the dual multilinguality property of MLP layers.
paper_ai_keywords: Direct Preference Optimization (DPO) multilingual Large Language Models (LLMs) zero-shot cross-lingual generalization mGPT-1.3B BLOOM Llama3 Aya-23 causal intervention activation analysis MLP layers bilingual sentence retrieval