title: DuoGuard: A Two-Player RL-Driven Framework for Multilingual LLM
    Guardrails
summary: The rapid advancement of large language models (LLMs) has increased the need
  for guardrail models to ensure responsible use, particularly in detecting
  unsafe and illegal content. While substantial safety data exist in English,
  multilingual guardrail modeling remains underexplored due to the scarcity of
  open-source safety data in other languages. To address this gap, we propose a
  novel two-player Reinforcement Learning (RL) framework, where a generator and a
  guardrail model co-evolve adversarially to produce high-quality synthetic data
  for multilingual guardrail training. We theoretically formalize this
  interaction as a two-player game, proving convergence to a Nash equilibrium.
  Empirical evaluations show that our model \ours outperforms state-of-the-art
  models, achieving nearly 10% improvement over LlamaGuard3 (8B) on English
  benchmarks while being 4.5x faster at inference with a significantly smaller
  model (0.5B). We achieve substantial advancements in multilingual safety tasks,
  particularly in addressing the imbalance for lower-resource languages in a
  collected real dataset. Ablation studies emphasize the critical role of
  synthetic data generation in bridging the imbalance in open-source data between
  English and other languages. These findings establish a scalable and efficient
  approach to synthetic data generation, paving the way for improved multilingual
  guardrail models to enhance LLM safety. Code, model, and data will be
  open-sourced at https://github.com/yihedeng9/DuoGuard.
paper_authors._id: 67a9604851169a582d14c113
paper_authors.name: Yihe Deng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-10T09:50:06.136Z
paper_authors._id: 642f4c789b2484d7d8551a93
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642f4c789b2484d7d8551a93/0lH4YXcbZa-Xlzj6ESo7F.jpeg
paper_authors.fullname: Yihe Deng
paper_authors.type: user
paper_authors.user: ydeng9
paper_authors._id: 67a9604851169a582d14c114
paper_authors.name: Yu Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T10:01:45.439Z
paper_authors._id: 62f82e52870a3f98bbf9e302
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f82e52870a3f98bbf9e302/5pN3oNBouZWlYu-uKa7lA.jpeg
paper_authors.fullname: Yu Yang
paper_authors.type: user
paper_authors.user: yuyangy
paper_authors._id: 67a9604851169a582d14c115
paper_authors.name: Junkai Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-10T16:11:40.494Z
paper_authors._id: 64e7bb81b159a6f87be99459
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e7bb81b159a6f87be99459/cxvzoEHg1YATnPJ9d3PTg.jpeg
paper_authors.fullname: Junkai Zhang
paper_authors.type: user
paper_authors.user: JunkaiZ
paper_authors._id: 67a9604851169a582d14c116
paper_authors.name: Wei Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-10T16:11:53.569Z
paper_authors._id: 62fa0ffe0697d224219a0cb7
paper_authors.avatarUrl: /avatars/f0ef59e1c0cf4ab4fe5cee08d488bd03.svg
paper_authors.fullname: Wei Wang
paper_authors.type: user
paper_authors.user: WeiWang
paper_authors._id: 67a9604851169a582d14c117
paper_authors.name: Bo Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-10T16:12:08.449Z
paper_authors._id: 6493236b70d925ae8050a1bf
paper_authors.avatarUrl: /avatars/b16069de1445cfa8608567175deaa2ae.svg
paper_authors.fullname: Bo Li
paper_authors.type: user
paper_authors.user: BoLi-aisecure
paper_title: DuoGuard: A Two-Player RL-Driven Framework for Multilingual LLM
    Guardrails
paper_summary: The rapid advancement of large language models (LLMs) has increased the need
  for guardrail models to ensure responsible use, particularly in detecting
  unsafe and illegal content. While substantial safety data exist in English,
  multilingual guardrail modeling remains underexplored due to the scarcity of
  open-source safety data in other languages. To address this gap, we propose a
  novel two-player Reinforcement Learning (RL) framework, where a generator and a
  guardrail model co-evolve adversarially to produce high-quality synthetic data
  for multilingual guardrail training. We theoretically formalize this
  interaction as a two-player game, proving convergence to a Nash equilibrium.
  Empirical evaluations show that our model \ours outperforms state-of-the-art
  models, achieving nearly 10% improvement over LlamaGuard3 (8B) on English
  benchmarks while being 4.5x faster at inference with a significantly smaller
  model (0.5B). We achieve substantial advancements in multilingual safety tasks,
  particularly in addressing the imbalance for lower-resource languages in a
  collected real dataset. Ablation studies emphasize the critical role of
  synthetic data generation in bridging the imbalance in open-source data between
  English and other languages. These findings establish a scalable and efficient
  approach to synthetic data generation, paving the way for improved multilingual
  guardrail models to enhance LLM safety. Code, model, and data will be
  open-sourced at https://github.com/yihedeng9/DuoGuard.
paper_ai_keywords: Reinforcement Learning (RL) generator metricrail model Nash equilibrium synthetic data multilingual guardrail training LlamaGuard3 multilingual safety tasks ablation studies synthetic data generation