Safe Unlearning: A Surprisingly Effective and Generalizable Solution to
  Defend Against Jailbreak Attacks
LLMs are known to be vulnerable to jailbreak attacks, even after safety
alignment. An important observation is that, while different types of jailbreak
attacks can generate significantly different queries, they mostly result in
similar responses that are rooted in the same harmful knowledge (e.g., detailed
steps to make a bomb). Therefore, we conjecture that directly unlearn the
harmful knowledge in the LLM can be a more effective way to defend against
jailbreak attacks than the mainstream supervised fine-tuning (SFT) based
approaches. Our extensive experiments confirmed our insight and suggested
surprising generalizability of our unlearning-based approach: using only 20 raw
harmful questions without any jailbreak prompt during training, our
solution reduced the Attack Success Rate (ASR) in Vicuna-7B on
out-of-distribution (OOD) harmful questions wrapped with various complex
jailbreak prompts from 82.6\% to 7.7\%. This significantly outperforms
Llama2-7B-Chat, which is fine-tuned on about 0.1M safety alignment samples but
still has an ASR of 21.9\% even under the help of an additional safety system
prompt. Further analysis reveals that the generalization ability of our
solution stems from the intrinsic relatedness among harmful responses across
harmful questions (e.g., response patterns, shared steps and actions, and
similarity among their learned representations in the LLM). Our code is
available at https://github.com/thu-coai/SafeUnlearning.
668a8f6517212ba35e0f77ec
Zhexin Zhang
claimed_verified
2024-07-08T08:50:09.660Z
61b58aa0d65058ce70beb98c
/avatars/aefd9271b891abc6dd2ded1a30eebca4.svg
Zhexin Zhang
user
nonstopfor
668a8f6517212ba35e0f77ed
Junxiao Yang
claimed_verified
2024-07-09T07:42:13.093Z
65d859a3661492b25c46a117
/avatars/06a547a00b472f512a25eef4ddd047bf.svg
Junxiao Yang
user
yangjunxiao2021
668a8f6517212ba35e0f77ee
Pei Ke
admin_assigned
2024-07-08T21:15:44.477Z
606c28fff3b1db0eeac26495
/avatars/edad72f14661a06367f360b7fedf3e1b.svg
Pei Ke
user
kepei1106
668a8f6517212ba35e0f77ef
Shiyao Cui
admin_assigned
2024-07-08T21:15:50.442Z
6617a9c192abaae4ecd5c066
/avatars/64994d014fac7f84e2925e568cc047f8.svg
Shiyao Cui
user
ShiyaoCui
668a8f6517212ba35e0f77f0
Chujie Zheng
admin_assigned
2024-07-08T21:18:54.726Z
610b70452719facd4ea85e28
https://cdn-avatars.huggingface.co/v1/production/uploads/610b70452719facd4ea85e28/S7nMy7D0Rxq0VIVblhYDG.jpeg
Chujie Zheng
user
chujiezheng
668a8f6517212ba35e0f77f1
Hongning Wang
668a8f6517212ba35e0f77f2
Minlie Huang
Safe Unlearning: A Surprisingly Effective and Generalizable Solution to
  Defend Against Jailbreak Attacks
LLMs are known to be vulnerable to jailbreak attacks, even after safety
alignment. An important observation is that, while different types of jailbreak
attacks can generate significantly different queries, they mostly result in
similar responses that are rooted in the same harmful knowledge (e.g., detailed
steps to make a bomb). Therefore, we conjecture that directly unlearn the
harmful knowledge in the LLM can be a more effective way to defend against
jailbreak attacks than the mainstream supervised fine-tuning (SFT) based
approaches. Our extensive experiments confirmed our insight and suggested
surprising generalizability of our unlearning-based approach: using only 20 raw
harmful questions without any jailbreak prompt during training, our
solution reduced the Attack Success Rate (ASR) in Vicuna-7B on
out-of-distribution (OOD) harmful questions wrapped with various complex
jailbreak prompts from 82.6\% to 7.7\%. This significantly outperforms
Llama2-7B-Chat, which is fine-tuned on about 0.1M safety alignment samples but
still has an ASR of 21.9\% even under the help of an additional safety system
prompt. Further analysis reveals that the generalization ability of our
solution stems from the intrinsic relatedness among harmful responses across
harmful questions (e.g., response patterns, shared steps and actions, and
similarity among their learned representations in the LLM). Our code is
available at https://github.com/thu-coai/SafeUnlearning.
An unlearning approach in LLMs was found to effectively reduce the success rate of jailbreak attacks by targeting harmful knowledge, which generalized better than supervised fine-tuning.
jailbreak attacks safe alignment harmful knowledge unlearning Vicuna-7B out-of-distribution attack success rate Llama2-7B-Chat harmful questions response patterns