title: Jailbroken: How Does LLM Safety Training Fail?
summary: Large language models trained for safety and harmlessness remain susceptible
  to adversarial misuse, as evidenced by the prevalence of "jailbreak" attacks on
  early releases of ChatGPT that elicit undesired behavior. Going beyond
  recognition of the issue, we investigate why such attacks succeed and how they
  can be created. We hypothesize two failure modes of safety training: competing
  objectives and mismatched generalization. Competing objectives arise when a
  model's capabilities and safety goals conflict, while mismatched generalization
  occurs when safety training fails to generalize to a domain for which
  capabilities exist. We use these failure modes to guide jailbreak design and
  then evaluate state-of-the-art models, including OpenAI's GPT-4 and Anthropic's
  Claude v1.3, against both existing and newly designed attacks. We find that
  vulnerabilities persist despite the extensive red-teaming and safety-training
  efforts behind these models. Notably, new attacks utilizing our failure modes
  succeed on every prompt in a collection of unsafe requests from the models'
  red-teaming evaluation sets and outperform existing ad hoc jailbreaks. Our
  analysis emphasizes the need for safety-capability parity -- that safety
  mechanisms should be as sophisticated as the underlying model -- and argues
  against the idea that scaling alone can resolve these safety failure modes.
paper_authors._id: 64a61ac1975e06b97eb3ccc0
paper_authors.name: Alexander Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-06T14:43:10.021Z
paper_authors._id: 64a6bf1d9e63efa992d45fe2
paper_authors.avatarUrl: /avatars/dffef2d43b18c2d5906b284379eda635.svg
paper_authors.fullname: Alexander Wei
paper_authors.type: user
paper_authors.user: aw31
paper_authors._id: 64a61ac1975e06b97eb3ccc1
paper_authors.name: Nika Haghtalab
paper_authors._id: 64a61ac1975e06b97eb3ccc2
paper_authors.name: Jacob Steinhardt
paper_title: Jailbroken: How Does LLM Safety Training Fail?
paper_summary: Large language models trained for safety and harmlessness remain susceptible
  to adversarial misuse, as evidenced by the prevalence of "jailbreak" attacks on
  early releases of ChatGPT that elicit undesired behavior. Going beyond
  recognition of the issue, we investigate why such attacks succeed and how they
  can be created. We hypothesize two failure modes of safety training: competing
  objectives and mismatched generalization. Competing objectives arise when a
  model's capabilities and safety goals conflict, while mismatched generalization
  occurs when safety training fails to generalize to a domain for which
  capabilities exist. We use these failure modes to guide jailbreak design and
  then evaluate state-of-the-art models, including OpenAI's GPT-4 and Anthropic's
  Claude v1.3, against both existing and newly designed attacks. We find that
  vulnerabilities persist despite the extensive red-teaming and safety-training
  efforts behind these models. Notably, new attacks utilizing our failure modes
  succeed on every prompt in a collection of unsafe requests from the models'
  red-teaming evaluation sets and outperform existing ad hoc jailbreaks. Our
  analysis emphasizes the need for safety-capability parity -- that safety
  mechanisms should be as sophisticated as the underlying model -- and argues
  against the idea that scaling alone can resolve these safety failure modes.
paper_ai_summary: Research identifies failure modes in safety training of large language models, leading to the creation of effective jailbreak attacks that exploit these vulnerabilities.
paper_ai_keywords: Large language models safety training adversarial misuse jailbreak attacks competing objectives mismatched generalization red-teaming evaluation sets safety-capability parity