title: Safe Unlearning: A Surprisingly Effective and Generalizable Solution to
    Defend Against Jailbreak Attacks
summary: LLMs are known to be vulnerable to jailbreak attacks, even after safety
  alignment. An important observation is that, while different types of jailbreak
  attacks can generate significantly different queries, they mostly result in
  similar responses that are rooted in the same harmful knowledge (e.g., detailed
  steps to make a bomb). Therefore, we conjecture that directly unlearn the
  harmful knowledge in the LLM can be a more effective way to defend against
  jailbreak attacks than the mainstream supervised fine-tuning (SFT) based
  approaches. Our extensive experiments confirmed our insight and suggested
  surprising generalizability of our unlearning-based approach: using only 20 raw
  harmful questions without any jailbreak prompt during training, our
  solution reduced the Attack Success Rate (ASR) in Vicuna-7B on
  out-of-distribution (OOD) harmful questions wrapped with various complex
  jailbreak prompts from 82.6\% to 7.7\%. This significantly outperforms
  Llama2-7B-Chat, which is fine-tuned on about 0.1M safety alignment samples but
  still has an ASR of 21.9\% even under the help of an additional safety system
  prompt. Further analysis reveals that the generalization ability of our
  solution stems from the intrinsic relatedness among harmful responses across
  harmful questions (e.g., response patterns, shared steps and actions, and
  similarity among their learned representations in the LLM). Our code is
  available at https://github.com/thu-coai/SafeUnlearning.
paper_authors._id: 668a8f6517212ba35e0f77ec
paper_authors.name: Zhexin Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-08T08:50:09.660Z
paper_authors._id: 61b58aa0d65058ce70beb98c
paper_authors.avatarUrl: /avatars/aefd9271b891abc6dd2ded1a30eebca4.svg
paper_authors.fullname: Zhexin Zhang
paper_authors.type: user
paper_authors.user: nonstopfor
paper_authors._id: 668a8f6517212ba35e0f77ed
paper_authors.name: Junxiao Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-09T07:42:13.093Z
paper_authors._id: 65d859a3661492b25c46a117
paper_authors.avatarUrl: /avatars/06a547a00b472f512a25eef4ddd047bf.svg
paper_authors.fullname: junxiao yang
paper_authors.type: user
paper_authors.user: yangjunxiao2021
paper_authors._id: 668a8f6517212ba35e0f77ee
paper_authors.name: Pei Ke
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-08T21:15:44.477Z
paper_authors._id: 606c28fff3b1db0eeac26495
paper_authors.avatarUrl: /avatars/edad72f14661a06367f360b7fedf3e1b.svg
paper_authors.fullname: Pei Ke
paper_authors.type: user
paper_authors.user: kepei1106
paper_authors._id: 668a8f6517212ba35e0f77ef
paper_authors.name: Shiyao Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-08T21:15:50.442Z
paper_authors._id: 6617a9c192abaae4ecd5c066
paper_authors.avatarUrl: /avatars/64994d014fac7f84e2925e568cc047f8.svg
paper_authors.fullname: Shiyao Cui
paper_authors.type: user
paper_authors.user: ShiyaoCui
paper_authors._id: 668a8f6517212ba35e0f77f0
paper_authors.name: Chujie Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-08T21:18:54.726Z
paper_authors._id: 610b70452719facd4ea85e28
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/610b70452719facd4ea85e28/S7nMy7D0Rxq0VIVblhYDG.jpeg
paper_authors.fullname: Chujie Zheng
paper_authors.type: user
paper_authors.user: chujiezheng
paper_authors._id: 668a8f6517212ba35e0f77f1
paper_authors.name: Hongning Wang
paper_authors._id: 668a8f6517212ba35e0f77f2
paper_authors.name: Minlie Huang
paper_title: Safe Unlearning: A Surprisingly Effective and Generalizable Solution to
    Defend Against Jailbreak Attacks
paper_summary: LLMs are known to be vulnerable to jailbreak attacks, even after safety
  alignment. An important observation is that, while different types of jailbreak
  attacks can generate significantly different queries, they mostly result in
  similar responses that are rooted in the same harmful knowledge (e.g., detailed
  steps to make a bomb). Therefore, we conjecture that directly unlearn the
  harmful knowledge in the LLM can be a more effective way to defend against
  jailbreak attacks than the mainstream supervised fine-tuning (SFT) based
  approaches. Our extensive experiments confirmed our insight and suggested
  surprising generalizability of our unlearning-based approach: using only 20 raw
  harmful questions without any jailbreak prompt during training, our
  solution reduced the Attack Success Rate (ASR) in Vicuna-7B on
  out-of-distribution (OOD) harmful questions wrapped with various complex
  jailbreak prompts from 82.6\% to 7.7\%. This significantly outperforms
  Llama2-7B-Chat, which is fine-tuned on about 0.1M safety alignment samples but
  still has an ASR of 21.9\% even under the help of an additional safety system
  prompt. Further analysis reveals that the generalization ability of our
  solution stems from the intrinsic relatedness among harmful responses across
  harmful questions (e.g., response patterns, shared steps and actions, and
  similarity among their learned representations in the LLM). Our code is
  available at https://github.com/thu-coai/SafeUnlearning.
paper_ai_summary: An unlearning approach in LLMs was found to effectively reduce the success rate of jailbreak attacks by targeting harmful knowledge, which generalized better than supervised fine-tuning.
paper_ai_keywords: jailbreak attacks safe alignment harmful knowledge unlearning Vicuna-7B out-of-distribution attack success rate Llama2-7B-Chat harmful questions response patterns