WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks,
  and Refusals of LLMs
We introduce WildGuard -- an open, light-weight moderation tool for LLM
safety that achieves three goals: (1) identifying malicious intent in user
prompts, (2) detecting safety risks of model responses, and (3) determining
model refusal rate. Together, WildGuard serves the increasing needs for
automatic safety moderation and evaluation of LLM interactions, providing a
one-stop tool with enhanced accuracy and broad coverage across 13 risk
categories. While existing open moderation tools such as Llama-Guard2 score
reasonably well in classifying straightforward model interactions, they lag far
behind a prompted GPT-4, especially in identifying adversarial jailbreaks and
in evaluating models' refusals, a key measure for evaluating safety behaviors
in model responses.
  To address these challenges, we construct WildGuardMix, a large-scale and
carefully balanced multi-task safety moderation dataset with 92K labeled
examples that cover vanilla (direct) prompts and adversarial jailbreaks, paired
with various refusal and compliance responses. WildGuardMix is a combination of
WildGuardTrain, the training data of WildGuard, and WildGuardTest, a
high-quality human-annotated moderation test set with 5K labeled items covering
broad risk scenarios. Through extensive evaluations on WildGuardTest and ten
existing public benchmarks, we show that WildGuard establishes state-of-the-art
performance in open-source safety moderation across all the three tasks
compared to ten strong existing open-source moderation models (e.g., up to
26.4% improvement on refusal detection). Importantly, WildGuard matches and
sometimes exceeds GPT-4 performance (e.g., up to 3.9% improvement on prompt
harmfulness identification). WildGuard serves as a highly effective safety
moderator in an LLM interface, reducing the success rate of jailbreak attacks
from 79.8% to 2.4%.
667cfeebaeba4a9f63a212f8
Seungju Han
extracted_pending
2024-06-27T05:55:56.668Z
666d0acab955b0e6553295e3
/avatars/51cf727e1037c2efd91c6276b56afaf6.svg
Seungju Han
user
seungjuh
667cfeebaeba4a9f63a212f9
Kavel Rao
667cfeebaeba4a9f63a212fa
Allyson Ettinger
claimed_verified
2024-10-31T09:55:34.119Z
6500b18c9d5687f825518282
/avatars/2abc8340eae436253b8b835eb8cfa505.svg
Allyson Ettinger
user
aettinger
667cfeebaeba4a9f63a212fb
Liwei Jiang
667cfeebaeba4a9f63a212fc
Bill Yuchen Lin
claimed_verified
2024-06-27T07:31:01.114Z
607f666a4ad99100d63ce35c
https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
Bill Yuchen Lin
user
yuchenlin
667cfeebaeba4a9f63a212fd
Nathan Lambert
claimed_verified
2025-01-06T08:01:08.260Z
628e5f90a9a3c754c1f7c88f
https://cdn-avatars.huggingface.co/v1/production/uploads/628e5f90a9a3c754c1f7c88f/iWqMY_l6dalrgRaJZWbK3.png
Nathan Lambert
user
natolambert
667cfeebaeba4a9f63a212fe
Yejin Choi
667cfeebaeba4a9f63a212ff
Nouha Dziri
extracted_pending
2024-06-27T05:55:56.668Z
653152a67aa28f32b0397d38
/avatars/2f0fc0abe38bc8cf996f9c8c7e90c46f.svg
Nouha Dziri
user
nouhad
WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks,
  and Refusals of LLMs
We introduce WildGuard -- an open, light-weight moderation tool for LLM
safety that achieves three goals: (1) identifying malicious intent in user
prompts, (2) detecting safety risks of model responses, and (3) determining
model refusal rate. Together, WildGuard serves the increasing needs for
automatic safety moderation and evaluation of LLM interactions, providing a
one-stop tool with enhanced accuracy and broad coverage across 13 risk
categories. While existing open moderation tools such as Llama-Guard2 score
reasonably well in classifying straightforward model interactions, they lag far
behind a prompted GPT-4, especially in identifying adversarial jailbreaks and
in evaluating models' refusals, a key measure for evaluating safety behaviors
in model responses.
  To address these challenges, we construct WildGuardMix, a large-scale and
carefully balanced multi-task safety moderation dataset with 92K labeled
examples that cover vanilla (direct) prompts and adversarial jailbreaks, paired
with various refusal and compliance responses. WildGuardMix is a combination of
WildGuardTrain, the training data of WildGuard, and WildGuardTest, a
high-quality human-annotated moderation test set with 5K labeled items covering
broad risk scenarios. Through extensive evaluations on WildGuardTest and ten
existing public benchmarks, we show that WildGuard establishes state-of-the-art
performance in open-source safety moderation across all the three tasks
compared to ten strong existing open-source moderation models (e.g., up to
26.4% improvement on refusal detection). Importantly, WildGuard matches and
sometimes exceeds GPT-4 performance (e.g., up to 3.9% improvement on prompt
harmfulness identification). WildGuard serves as a highly effective safety
moderator in an LLM interface, reducing the success rate of jailbreak attacks
from 79.8% to 2.4%.
WildGuard is an open, lightweight moderation tool for LLM safety that identifies malicious intent, detects safety risks, and assesses refusal rates, achieving state-of-the-art performance across multiple benchmarks.
WildGuard LLM safety malicious intent safety risks model refusal rate WildGuardMix adversarial jailbreaks refusal detection prompt harmfulness identification