title: Jailbreaking to Jailbreak
summary: Refusal training on Large Language Models (LLMs) prevents harmful outputs,
  yet this defense remains vulnerable to both automated and human-crafted
  jailbreaks. We present a novel LLM-as-red-teamer approach in which a human
  jailbreaks a refusal-trained LLM to make it willing to jailbreak itself or
  other LLMs. We refer to the jailbroken LLMs as J_2 attackers, which can
  systematically evaluate target models using various red teaming strategies and
  improve its performance via in-context learning from the previous failures. Our
  experiments demonstrate that Sonnet 3.5 and Gemini 1.5 pro outperform other
  LLMs as J_2, achieving 93.0% and 91.0% attack success rates (ASRs)
  respectively against GPT-4o (and similar results across other capable LLMs) on
  Harmbench. Our work not only introduces a scalable approach to strategic red
  teaming, drawing inspiration from human red teamers, but also highlights
  jailbreaking-to-jailbreak as an overlooked failure mode of the safeguard.
  Specifically, an LLM can bypass its own safeguards by employing a jailbroken
  version of itself that is willing to assist in further jailbreaking. To prevent
  any direct misuse with J_2, while advancing research in AI safety, we
  publicly share our methodology while keeping specific prompting details
  private.
paper_authors._id: 67b2c3386ccf462ccaa45860
paper_authors.name: Jeremy Kritz
paper_authors._id: 67b2c3386ccf462ccaa45861
paper_authors.name: Vaughn Robinson
paper_authors._id: 67b2c3386ccf462ccaa45862
paper_authors.name: Robert Vacareanu
paper_authors._id: 67b2c3386ccf462ccaa45863
paper_authors.name: Bijan Varjavand
paper_authors._id: 67b2c3386ccf462ccaa45864
paper_authors.name: Michael Choi
paper_authors._id: 67b2c3386ccf462ccaa45865
paper_authors.name: Bobby Gogov
paper_authors._id: 67b2c3386ccf462ccaa45866
paper_authors.name: Scale Red Team
paper_authors._id: 67b2c3386ccf462ccaa45867
paper_authors.name: Summer Yue
paper_authors._id: 67b2c3386ccf462ccaa45868
paper_authors.name: Willow E. Primack
paper_authors._id: 67b2c3386ccf462ccaa45869
paper_authors.name: Zifan Wang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-17T05:03:53.788Z
paper_authors._id: 66976d1007b36ccd01586ce5
paper_authors.avatarUrl: /avatars/5811e350907a29b71f6e4d57ffd53e66.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: ZifanScale
paper_title: Jailbreaking to Jailbreak
paper_summary: Refusal training on Large Language Models (LLMs) prevents harmful outputs,
  yet this defense remains vulnerable to both automated and human-crafted
  jailbreaks. We present a novel LLM-as-red-teamer approach in which a human
  jailbreaks a refusal-trained LLM to make it willing to jailbreak itself or
  other LLMs. We refer to the jailbroken LLMs as J_2 attackers, which can
  systematically evaluate target models using various red teaming strategies and
  improve its performance via in-context learning from the previous failures. Our
  experiments demonstrate that Sonnet 3.5 and Gemini 1.5 pro outperform other
  LLMs as J_2, achieving 93.0% and 91.0% attack success rates (ASRs)
  respectively against GPT-4o (and similar results across other capable LLMs) on
  Harmbench. Our work not only introduces a scalable approach to strategic red
  teaming, drawing inspiration from human red teamers, but also highlights
  jailbreaking-to-jailbreak as an overlooked failure mode of the safeguard.
  Specifically, an LLM can bypass its own safeguards by employing a jailbroken
  version of itself that is willing to assist in further jailbreaking. To prevent
  any direct misuse with J_2, while advancing research in AI safety, we
  publicly share our methodology while keeping specific prompting details
  private.
paper_ai_keywords: Large Language Models (LLMs) Refusal training Jailbreaks LLM-as-red-teamer Automated jailbreaks Human-crafted jailbreaks Target models In-context learning Attack team about jailbreaking and evaluation Attack success rates (ASRs) Harmbench Strategic red teaming Multimodal representation Safeguards Jailbreaking-to-jailbreak GPT-4