title: MART: Improving LLM Safety with Multi-round Automatic Red-Teaming
summary: Red-teaming is a common practice for mitigating unsafe behaviors in Large
  Language Models (LLMs), which involves thoroughly assessing LLMs to identify
  potential flaws and addressing them with responsible and accurate responses.
  While effective, manual red-teaming is costly, and existing automatic
  red-teaming typically discovers safety risks without addressing them. In this
  paper, we propose a Multi-round Automatic Red-Teaming (MART) method, which
  incorporates both automatic adversarial prompt writing and safe response
  generation, significantly increasing red-teaming scalability and the safety of
  the target LLM. Specifically, an adversarial LLM and a target LLM interplay
  with each other in an iterative manner, where the adversarial LLM aims to
  generate challenging prompts that elicit unsafe responses from the target LLM,
  while the target LLM is fine-tuned with safety aligned data on these
  adversarial prompts. In each round, the adversarial LLM crafts better attacks
  on the updated target LLM, while the target LLM also improves itself through
  safety fine-tuning. On adversarial prompt benchmarks, the violation rate of an
  LLM with limited safety alignment reduces up to 84.7% after 4 rounds of MART,
  achieving comparable performance to LLMs with extensive adversarial prompt
  writing. Notably, model helpfulness on non-adversarial prompts remains stable
  throughout iterations, indicating the target LLM maintains strong performance
  on instruction following.
paper_authors._id: 65542e46ddcd3e36f416516b
paper_authors.name: Suyu Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T09:41:32.970Z
paper_authors._id: 64f400fd0ff19714e24caaef
paper_authors.avatarUrl: /avatars/a690ae5b30377b8179b45839447f7f93.svg
paper_authors.fullname: Suyu Ge
paper_authors.type: user
paper_authors.user: suuyu
paper_authors._id: 65542e46ddcd3e36f416516c
paper_authors.name: Chunting Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T09:43:02.669Z
paper_authors._id: 64f8d07dd493d8b0d2c04ecd
paper_authors.avatarUrl: /avatars/7112232dcadcd2a7c12ec635d781b3ea.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Chunting
paper_authors._id: 65542e46ddcd3e36f416516d
paper_authors.name: Rui Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T09:41:49.302Z
paper_authors._id: 6407d236a7bc7c3865adcc3a
paper_authors.avatarUrl: /avatars/8b75cc07fd049acde5363d268c5b88d4.svg
paper_authors.fullname: Rui Hou
paper_authors.type: user
paper_authors.user: rayhou
paper_authors._id: 65542e46ddcd3e36f416516e
paper_authors.name: Madian Khabsa
paper_authors._id: 65542e46ddcd3e36f416516f
paper_authors.name: Yi-Chia Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T09:42:00.449Z
paper_authors._id: 641b9775abfce26bcf7d0753
paper_authors.avatarUrl: /avatars/778e2f1bb7fd5a80bfcf3cfb6863aa4c.svg
paper_authors.fullname: Yi-Chia Wang
paper_authors.type: user
paper_authors.user: yichiawang
paper_authors._id: 65542e46ddcd3e36f4165170
paper_authors.name: Qifan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T09:42:11.404Z
paper_authors._id: 64b65c84c991daa61404abd7
paper_authors.avatarUrl: /avatars/3592aa8774ea4e2836099e0701b4b78d.svg
paper_authors.fullname: WangQifan
paper_authors.type: user
paper_authors.user: Bayees
paper_authors._id: 65542e46ddcd3e36f4165171
paper_authors.name: Jiawei Han
paper_authors._id: 65542e46ddcd3e36f4165172
paper_authors.name: Yuning Mao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T09:42:46.743Z
paper_authors._id: 62bf46a630a1888a761f8619
paper_authors.avatarUrl: /avatars/de6c656066b1e88d37882ce68a0fd25c.svg
paper_authors.fullname: Yuning Mao
paper_authors.type: user
paper_authors.user: yuningm
paper_title: MART: Improving LLM Safety with Multi-round Automatic Red-Teaming
paper_summary: Red-teaming is a common practice for mitigating unsafe behaviors in Large
  Language Models (LLMs), which involves thoroughly assessing LLMs to identify
  potential flaws and addressing them with responsible and accurate responses.
  While effective, manual red-teaming is costly, and existing automatic
  red-teaming typically discovers safety risks without addressing them. In this
  paper, we propose a Multi-round Automatic Red-Teaming (MART) method, which
  incorporates both automatic adversarial prompt writing and safe response
  generation, significantly increasing red-teaming scalability and the safety of
  the target LLM. Specifically, an adversarial LLM and a target LLM interplay
  with each other in an iterative manner, where the adversarial LLM aims to
  generate challenging prompts that elicit unsafe responses from the target LLM,
  while the target LLM is fine-tuned with safety aligned data on these
  adversarial prompts. In each round, the adversarial LLM crafts better attacks
  on the updated target LLM, while the target LLM also improves itself through
  safety fine-tuning. On adversarial prompt benchmarks, the violation rate of an
  LLM with limited safety alignment reduces up to 84.7% after 4 rounds of MART,
  achieving comparable performance to LLMs with extensive adversarial prompt
  writing. Notably, model helpfulness on non-adversarial prompts remains stable
  throughout iterations, indicating the target LLM maintains strong performance
  on instruction following.
paper_ai_keywords: adversarial prompt writing safe response generation Multi-roundAutomatic Red-Teaming (MART) adversarial LLM target LLM iterative manner adversarial prompts safety aligned data safety according-tuning violation rate adversarial prompt benchmarks non-adversarial prompts instruction following