title: WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks,
    and Refusals of LLMs
summary: We introduce WildGuard -- an open, light-weight moderation tool for LLM
  safety that achieves three goals: (1) identifying malicious intent in user
  prompts, (2) detecting safety risks of model responses, and (3) determining
  model refusal rate. Together, WildGuard serves the increasing needs for
  automatic safety moderation and evaluation of LLM interactions, providing a
  one-stop tool with enhanced accuracy and broad coverage across 13 risk
  categories. While existing open moderation tools such as Llama-Guard2 score
  reasonably well in classifying straightforward model interactions, they lag far
  behind a prompted GPT-4, especially in identifying adversarial jailbreaks and
  in evaluating models' refusals, a key measure for evaluating safety behaviors
  in model responses.
    To address these challenges, we construct WildGuardMix, a large-scale and
  carefully balanced multi-task safety moderation dataset with 92K labeled
  examples that cover vanilla (direct) prompts and adversarial jailbreaks, paired
  with various refusal and compliance responses. WildGuardMix is a combination of
  WildGuardTrain, the training data of WildGuard, and WildGuardTest, a
  high-quality human-annotated moderation test set with 5K labeled items covering
  broad risk scenarios. Through extensive evaluations on WildGuardTest and ten
  existing public benchmarks, we show that WildGuard establishes state-of-the-art
  performance in open-source safety moderation across all the three tasks
  compared to ten strong existing open-source moderation models (e.g., up to
  26.4% improvement on refusal detection). Importantly, WildGuard matches and
  sometimes exceeds GPT-4 performance (e.g., up to 3.9% improvement on prompt
  harmfulness identification). WildGuard serves as a highly effective safety
  moderator in an LLM interface, reducing the success rate of jailbreak attacks
  from 79.8% to 2.4%.
paper_authors._id: 667cfeebaeba4a9f63a212f8
paper_authors.name: Seungju Han
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-06-27T05:55:56.668Z
paper_authors._id: 666d0acab955b0e6553295e3
paper_authors.avatarUrl: /avatars/51cf727e1037c2efd91c6276b56afaf6.svg
paper_authors.fullname: Seungju Han
paper_authors.type: user
paper_authors.user: seungjuh
paper_authors._id: 667cfeebaeba4a9f63a212f9
paper_authors.name: Kavel Rao
paper_authors._id: 667cfeebaeba4a9f63a212fa
paper_authors.name: Allyson Ettinger
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-31T09:55:34.119Z
paper_authors._id: 6500b18c9d5687f825518282
paper_authors.avatarUrl: /avatars/2abc8340eae436253b8b835eb8cfa505.svg
paper_authors.fullname: Allyson Ettinger
paper_authors.type: user
paper_authors.user: aettinger
paper_authors._id: 667cfeebaeba4a9f63a212fb
paper_authors.name: Liwei Jiang
paper_authors._id: 667cfeebaeba4a9f63a212fc
paper_authors.name: Bill Yuchen Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-27T07:31:01.114Z
paper_authors._id: 607f666a4ad99100d63ce35c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
paper_authors.fullname: Bill Yuchen Lin
paper_authors.type: user
paper_authors.user: yuchenlin
paper_authors._id: 667cfeebaeba4a9f63a212fd
paper_authors.name: Nathan Lambert
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-06T08:01:08.260Z
paper_authors._id: 628e5f90a9a3c754c1f7c88f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/628e5f90a9a3c754c1f7c88f/iWqMY_l6dalrgRaJZWbK3.png
paper_authors.fullname: Nathan Lambert
paper_authors.type: user
paper_authors.user: natolambert
paper_authors._id: 667cfeebaeba4a9f63a212fe
paper_authors.name: Yejin Choi
paper_authors._id: 667cfeebaeba4a9f63a212ff
paper_authors.name: Nouha Dziri
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-06-27T05:55:56.668Z
paper_authors._id: 653152a67aa28f32b0397d38
paper_authors.avatarUrl: /avatars/2f0fc0abe38bc8cf996f9c8c7e90c46f.svg
paper_authors.fullname: Nouha Dziri
paper_authors.type: user
paper_authors.user: nouhad
paper_title: WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks,
    and Refusals of LLMs
paper_summary: We introduce WildGuard -- an open, light-weight moderation tool for LLM
  safety that achieves three goals: (1) identifying malicious intent in user
  prompts, (2) detecting safety risks of model responses, and (3) determining
  model refusal rate. Together, WildGuard serves the increasing needs for
  automatic safety moderation and evaluation of LLM interactions, providing a
  one-stop tool with enhanced accuracy and broad coverage across 13 risk
  categories. While existing open moderation tools such as Llama-Guard2 score
  reasonably well in classifying straightforward model interactions, they lag far
  behind a prompted GPT-4, especially in identifying adversarial jailbreaks and
  in evaluating models' refusals, a key measure for evaluating safety behaviors
  in model responses.
    To address these challenges, we construct WildGuardMix, a large-scale and
  carefully balanced multi-task safety moderation dataset with 92K labeled
  examples that cover vanilla (direct) prompts and adversarial jailbreaks, paired
  with various refusal and compliance responses. WildGuardMix is a combination of
  WildGuardTrain, the training data of WildGuard, and WildGuardTest, a
  high-quality human-annotated moderation test set with 5K labeled items covering
  broad risk scenarios. Through extensive evaluations on WildGuardTest and ten
  existing public benchmarks, we show that WildGuard establishes state-of-the-art
  performance in open-source safety moderation across all the three tasks
  compared to ten strong existing open-source moderation models (e.g., up to
  26.4% improvement on refusal detection). Importantly, WildGuard matches and
  sometimes exceeds GPT-4 performance (e.g., up to 3.9% improvement on prompt
  harmfulness identification). WildGuard serves as a highly effective safety
  moderator in an LLM interface, reducing the success rate of jailbreak attacks
  from 79.8% to 2.4%.
paper_ai_summary: WildGuard is an open, lightweight moderation tool for LLM safety that identifies malicious intent, detects safety risks, and assesses refusal rates, achieving state-of-the-art performance across multiple benchmarks.
paper_ai_keywords: WildGuard LLM safety malicious intent safety risks model refusal rate WildGuardMix adversarial jailbreaks refusal detection prompt harmfulness identification