title: Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts
summary: As large language models (LLMs) become increasingly prevalent across many
  real-world applications, understanding and enhancing their robustness to user
  inputs is of paramount importance. Existing methods for identifying adversarial
  prompts tend to focus on specific domains, lack diversity, or require extensive
  human annotations. To address these limitations, we present Rainbow Teaming, a
  novel approach for producing a diverse collection of adversarial prompts.
  Rainbow Teaming casts adversarial prompt generation as a quality-diversity
  problem, and uses open-ended search to generate prompts that are both effective
  and diverse. It can uncover a model's vulnerabilities across a broad range of
  domains including, in this paper, safety, question answering, and
  cybersecurity. We also demonstrate that fine-tuning on synthetic data generated
  by Rainbow Teaming improves the safety of state-of-the-art LLMs without hurting
  their general capabilities and helpfulness, paving the path to open-ended
  self-improvement.
paper_authors._id: 65dd759e89a2a760d1637c32
paper_authors.name: Mikayel Samvelyan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T10:42:57.674Z
paper_authors._id: 64f650648afa37b1248c8792
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f650648afa37b1248c8792/jqYcQWIdLrnBXASnVBUp-.jpeg
paper_authors.fullname: Mikayel Samvelyan
paper_authors.type: user
paper_authors.user: samvelyan
paper_authors._id: 65dd759e89a2a760d1637c33
paper_authors.name: Sharath Chandra Raparthy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T10:43:17.402Z
paper_authors._id: 65007a96ea5699b59a9bb432
paper_authors.avatarUrl: /avatars/4549f5964b007de189c45a77daf10075.svg
paper_authors.fullname: Sharath Raparthy
paper_authors.type: user
paper_authors.user: sharathraparthy
paper_authors._id: 65dd759e89a2a760d1637c34
paper_authors.name: Andrei Lupu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T10:43:23.914Z
paper_authors._id: 64c3ffbb17bd80604501fc28
paper_authors.avatarUrl: /avatars/cfe60c2c6b5656cb541357877aa757dc.svg
paper_authors.fullname: Andrei Lupu
paper_authors.type: user
paper_authors.user: iamexploit
paper_authors._id: 65dd759e89a2a760d1637c35
paper_authors.name: Eric Hambro
paper_authors._id: 65dd759e89a2a760d1637c36
paper_authors.name: Aram H. Markosyan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T10:43:36.607Z
paper_authors._id: 63fe54211193ba2c229bd74a
paper_authors.avatarUrl: /avatars/14c7cc8b9ebde8d6d4f16e578e311396.svg
paper_authors.fullname: Aram Markosyan
paper_authors.type: user
paper_authors.user: armarkos
paper_authors._id: 65dd759e89a2a760d1637c37
paper_authors.name: Manish Bhatt
paper_authors._id: 65dd759e89a2a760d1637c38
paper_authors.name: Yuning Mao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T10:44:17.726Z
paper_authors._id: 62bf46a630a1888a761f8619
paper_authors.avatarUrl: /avatars/de6c656066b1e88d37882ce68a0fd25c.svg
paper_authors.fullname: Yuning Mao
paper_authors.type: user
paper_authors.user: yuningm
paper_authors._id: 65dd759e89a2a760d1637c39
paper_authors.name: Minqi Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T10:45:23.233Z
paper_authors._id: 630368320547362a22a21798
paper_authors.avatarUrl: /avatars/035b468584e43b4fbbf4c4aa1d2bbac7.svg
paper_authors.fullname: Minqi Jiang
paper_authors.type: user
paper_authors.user: minqi
paper_authors._id: 65dd759e89a2a760d1637c3a
paper_authors.name: Jack Parker-Holder
paper_authors._id: 65dd759e89a2a760d1637c3b
paper_authors.name: Jakob Foerster
paper_authors._id: 65dd759e89a2a760d1637c3c
paper_authors.name: Tim Rocktäschel
paper_authors._id: 65dd759e89a2a760d1637c3d
paper_authors.name: Roberta Raileanu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T10:45:45.216Z
paper_authors._id: 633e94793a17ab61de8e2b9c
paper_authors.avatarUrl: /avatars/5f2f58ddeed211393660ada6b135f0d5.svg
paper_authors.fullname: Roberta Raileanu
paper_authors.type: user
paper_authors.user: rraileanu
paper_title: Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts
paper_summary: As large language models (LLMs) become increasingly prevalent across many
  real-world applications, understanding and enhancing their robustness to user
  inputs is of paramount importance. Existing methods for identifying adversarial
  prompts tend to focus on specific domains, lack diversity, or require extensive
  human annotations. To address these limitations, we present Rainbow Teaming, a
  novel approach for producing a diverse collection of adversarial prompts.
  Rainbow Teaming casts adversarial prompt generation as a quality-diversity
  problem, and uses open-ended search to generate prompts that are both effective
  and diverse. It can uncover a model's vulnerabilities across a broad range of
  domains including, in this paper, safety, question answering, and
  cybersecurity. We also demonstrate that fine-tuning on synthetic data generated
  by Rainbow Teaming improves the safety of state-of-the-art LLMs without hurting
  their general capabilities and helpfulness, paving the path to open-ended
  self-improvement.
paper_ai_keywords: adversarial prompts quality-diversity problem open-ended search fine-tuning synthetic data self-improvement