DuoGuard: A Two-Player RL-Driven Framework for Multilingual LLM
  Guardrails
The rapid advancement of large language models (LLMs) has increased the need
for guardrail models to ensure responsible use, particularly in detecting
unsafe and illegal content. While substantial safety data exist in English,
multilingual guardrail modeling remains underexplored due to the scarcity of
open-source safety data in other languages. To address this gap, we propose a
novel two-player Reinforcement Learning (RL) framework, where a generator and a
guardrail model co-evolve adversarially to produce high-quality synthetic data
for multilingual guardrail training. We theoretically formalize this
interaction as a two-player game, proving convergence to a Nash equilibrium.
Empirical evaluations show that our model \ours outperforms state-of-the-art
models, achieving nearly 10% improvement over LlamaGuard3 (8B) on English
benchmarks while being 4.5x faster at inference with a significantly smaller
model (0.5B). We achieve substantial advancements in multilingual safety tasks,
particularly in addressing the imbalance for lower-resource languages in a
collected real dataset. Ablation studies emphasize the critical role of
synthetic data generation in bridging the imbalance in open-source data between
English and other languages. These findings establish a scalable and efficient
approach to synthetic data generation, paving the way for improved multilingual
guardrail models to enhance LLM safety. Code, model, and data will be
open-sourced at https://github.com/yihedeng9/DuoGuard.
67a9604851169a582d14c113
Yihe Deng
claimed_verified
2025-02-10T09:50:06.136Z
642f4c789b2484d7d8551a93
https://cdn-avatars.huggingface.co/v1/production/uploads/642f4c789b2484d7d8551a93/0lH4YXcbZa-Xlzj6ESo7F.jpeg
Yihe Deng
user
ydeng9
67a9604851169a582d14c114
Yu Yang
claimed_verified
2025-02-11T10:01:45.439Z
62f82e52870a3f98bbf9e302
https://cdn-avatars.huggingface.co/v1/production/uploads/62f82e52870a3f98bbf9e302/5pN3oNBouZWlYu-uKa7lA.jpeg
Yu Yang
user
yuyangy
67a9604851169a582d14c115
Junkai Zhang
admin_assigned
2025-02-10T16:11:40.494Z
64e7bb81b159a6f87be99459
https://cdn-avatars.huggingface.co/v1/production/uploads/64e7bb81b159a6f87be99459/obeVMow9SRRNt722T6ZvU.jpeg
Junkai Zhang
user
JunkaiZ
67a9604851169a582d14c116
Wei Wang
admin_assigned
2025-02-10T16:11:53.569Z
62fa0ffe0697d224219a0cb7
/avatars/f0ef59e1c0cf4ab4fe5cee08d488bd03.svg
Wei Wang
user
WeiWang
67a9604851169a582d14c117
Bo Li
admin_assigned
2025-02-10T16:12:08.449Z
6493236b70d925ae8050a1bf
/avatars/b16069de1445cfa8608567175deaa2ae.svg
Bo Li
user
BoLi-aisecure
DuoGuard: A Two-Player RL-Driven Framework for Multilingual LLM
  Guardrails
The rapid advancement of large language models (LLMs) has increased the need
for guardrail models to ensure responsible use, particularly in detecting
unsafe and illegal content. While substantial safety data exist in English,
multilingual guardrail modeling remains underexplored due to the scarcity of
open-source safety data in other languages. To address this gap, we propose a
novel two-player Reinforcement Learning (RL) framework, where a generator and a
guardrail model co-evolve adversarially to produce high-quality synthetic data
for multilingual guardrail training. We theoretically formalize this
interaction as a two-player game, proving convergence to a Nash equilibrium.
Empirical evaluations show that our model \ours outperforms state-of-the-art
models, achieving nearly 10% improvement over LlamaGuard3 (8B) on English
benchmarks while being 4.5x faster at inference with a significantly smaller
model (0.5B). We achieve substantial advancements in multilingual safety tasks,
particularly in addressing the imbalance for lower-resource languages in a
collected real dataset. Ablation studies emphasize the critical role of
synthetic data generation in bridging the imbalance in open-source data between
English and other languages. These findings establish a scalable and efficient
approach to synthetic data generation, paving the way for improved multilingual
guardrail models to enhance LLM safety. Code, model, and data will be
open-sourced at https://github.com/yihedeng9/DuoGuard.
A two-player adversarial reinforcement learning framework generates high-quality synthetic data for multilingual guardrail models, improving safety and efficiency across languages.
Reinforcement Learning RL generator guardrail model Nash equilibrium synthetic data generation multilingual safety LlamaGuard3