title: WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially)
    Safer Language Models
summary: We introduce WildTeaming, an automatic LLM safety red-teaming framework that
  mines in-the-wild user-chatbot interactions to discover 5.7K unique clusters of
  novel jailbreak tactics, and then composes multiple tactics for systematic
  exploration of novel jailbreaks. Compared to prior work that performed
  red-teaming via recruited human workers, gradient-based optimization, or
  iterative revision with LLMs, our work investigates jailbreaks from chatbot
  users who were not specifically instructed to break the system. WildTeaming
  reveals previously unidentified vulnerabilities of frontier LLMs, resulting in
  up to 4.6x more diverse and successful adversarial attacks compared to
  state-of-the-art jailbreak methods.
    While many datasets exist for jailbreak evaluation, very few open-source
  datasets exist for jailbreak training, as safety training data has been closed
  even when model weights are open. With WildTeaming we create WildJailbreak, a
  large-scale open-source synthetic safety dataset with 262K vanilla (direct
  request) and adversarial (complex jailbreak) prompt-response pairs. To mitigate
  exaggerated safety behaviors, WildJailbreak provides two contrastive types of
  queries: 1) harmful queries (vanilla & adversarial) and 2) benign queries that
  resemble harmful queries in form but contain no harm. As WildJailbreak
  considerably upgrades the quality and scale of existing safety resources, it
  uniquely enables us to examine the scaling effects of data and the interplay of
  data properties and model capabilities during safety training. Through
  extensive experiments, we identify the training properties that enable an ideal
  balance of safety behaviors: appropriate safeguarding without over-refusal,
  effective handling of vanilla and adversarial queries, and minimal, if any,
  decrease in general capabilities. All components of WildJailbeak contribute to
  achieving balanced safety behaviors of models.
paper_authors._id: 667d935aaf1461e67e477798
paper_authors.name: Liwei Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T18:56:52.798Z
paper_authors._id: 5efe5c89ff69163f6f59e628
paper_authors.avatarUrl: /avatars/c7ea475c50fcce528c9b748f279058eb.svg
paper_authors.fullname: Liwei Jiang
paper_authors.type: user
paper_authors.user: liweijiang
paper_authors._id: 667d935aaf1461e67e477799
paper_authors.name: Kavel Rao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T18:57:00.770Z
paper_authors._id: 6530cf185da070af77b69484
paper_authors.avatarUrl: /avatars/44836b07fd5f9a0be3348e01267152a0.svg
paper_authors.fullname: Kavel Rao
paper_authors.type: user
paper_authors.user: kavelrao
paper_authors._id: 667d935aaf1461e67e47779a
paper_authors.name: Seungju Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T18:57:11.745Z
paper_authors._id: 614c569f9691b6ffd13ff2ca
paper_authors.avatarUrl: /avatars/49601b76b8a13136fe140161e455b7c7.svg
paper_authors.fullname: Seungju Han
paper_authors.type: user
paper_authors.user: Seungjuhan
paper_authors._id: 667d935aaf1461e67e47779b
paper_authors.name: Allyson Ettinger
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T18:57:18.330Z
paper_authors._id: 6500b18c9d5687f825518282
paper_authors.avatarUrl: /avatars/2abc8340eae436253b8b835eb8cfa505.svg
paper_authors.fullname: Allyson Ettinger
paper_authors.type: user
paper_authors.user: aettinger
paper_authors._id: 667d935aaf1461e67e47779c
paper_authors.name: Faeze Brahman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T18:57:24.452Z
paper_authors._id: 65282b8d578679aac7888aec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65282b8d578679aac7888aec/dibBkhH-z1c70mJZZxJ7u.jpeg
paper_authors.fullname: Faeze Brahman
paper_authors.type: user
paper_authors.user: faezeb
paper_authors._id: 667d935aaf1461e67e47779d
paper_authors.name: Sachin Kumar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T18:57:35.335Z
paper_authors._id: 63f24d2d7ddf724fbcc0ea9c
paper_authors.avatarUrl: /avatars/3e24c1aa9c1b4066d2dd56aeb4b0f62e.svg
paper_authors.fullname: sachin kumar
paper_authors.type: user
paper_authors.user: sachinkumar
paper_authors._id: 667d935aaf1461e67e47779e
paper_authors.name: Niloofar Mireshghallah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T18:57:50.130Z
paper_authors._id: 642154542cc2b3c39e7f47e2
paper_authors.avatarUrl: /avatars/78fa026109ff7cc96726e6e3cd581587.svg
paper_authors.fullname: Fatemeh  Mireshghallah
paper_authors.type: user
paper_authors.user: niloofarm
paper_authors._id: 667d935aaf1461e67e47779f
paper_authors.name: Ximing Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T18:57:57.608Z
paper_authors._id: 640928bd3461c51cf7378707
paper_authors.avatarUrl: /avatars/429d1ba8327df82a8f8360c0f55e199c.svg
paper_authors.fullname: Ximing Lu
paper_authors.type: user
paper_authors.user: Ximing
paper_authors._id: 667d935aaf1461e67e4777a0
paper_authors.name: Maarten Sap
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T18:58:02.732Z
paper_authors._id: 62e0657168d80187d20e7b0e
paper_authors.avatarUrl: /avatars/d8de3ec3edc381b34e6f908eeeea86ee.svg
paper_authors.fullname: Maarten Sap
paper_authors.type: user
paper_authors.user: maartensap
paper_authors._id: 667d935aaf1461e67e4777a1
paper_authors.name: Yejin Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T18:58:11.100Z
paper_authors._id: 64d42729f63b01b7f676b176
paper_authors.avatarUrl: /avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
paper_authors.fullname: Yejin Choi
paper_authors.type: user
paper_authors.user: yejinchoinka
paper_authors._id: 667d935aaf1461e67e4777a2
paper_authors.name: Nouha Dziri
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-06-27T16:29:15.580Z
paper_authors._id: 653152a67aa28f32b0397d38
paper_authors.avatarUrl: /avatars/2f0fc0abe38bc8cf996f9c8c7e90c46f.svg
paper_authors.fullname: Nouha Dziri
paper_authors.type: user
paper_authors.user: nouhad
paper_title: WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially)
    Safer Language Models
paper_summary: We introduce WildTeaming, an automatic LLM safety red-teaming framework that
  mines in-the-wild user-chatbot interactions to discover 5.7K unique clusters of
  novel jailbreak tactics, and then composes multiple tactics for systematic
  exploration of novel jailbreaks. Compared to prior work that performed
  red-teaming via recruited human workers, gradient-based optimization, or
  iterative revision with LLMs, our work investigates jailbreaks from chatbot
  users who were not specifically instructed to break the system. WildTeaming
  reveals previously unidentified vulnerabilities of frontier LLMs, resulting in
  up to 4.6x more diverse and successful adversarial attacks compared to
  state-of-the-art jailbreak methods.
    While many datasets exist for jailbreak evaluation, very few open-source
  datasets exist for jailbreak training, as safety training data has been closed
  even when model weights are open. With WildTeaming we create WildJailbreak, a
  large-scale open-source synthetic safety dataset with 262K vanilla (direct
  request) and adversarial (complex jailbreak) prompt-response pairs. To mitigate
  exaggerated safety behaviors, WildJailbreak provides two contrastive types of
  queries: 1) harmful queries (vanilla & adversarial) and 2) benign queries that
  resemble harmful queries in form but contain no harm. As WildJailbreak
  considerably upgrades the quality and scale of existing safety resources, it
  uniquely enables us to examine the scaling effects of data and the interplay of
  data properties and model capabilities during safety training. Through
  extensive experiments, we identify the training properties that enable an ideal
  balance of safety behaviors: appropriate safeguarding without over-refusal,
  effective handling of vanilla and adversarial queries, and minimal, if any,
  decrease in general capabilities. All components of WildJailbeak contribute to
  achieving balanced safety behaviors of models.
paper_ai_keywords: large-scale open-source synthetic safety dataset adversarial prompt-response pairs contrastive types of queries harmful queries benign queries jailbreak tactics red-teaming gradient-based optimization iterative revision adversarial attacks safety training frontier LLMs exaggerated safety behaviors over-refusal