Weak-to-Strong Jailbreaking on Large Language Models
Although significant efforts have been dedicated to aligning large language
models (LLMs), red-teaming reports suggest that these carefully aligned LLMs
could still be jailbroken through adversarial prompts, tuning, or decoding.
Upon examining the jailbreaking vulnerability of aligned LLMs, we observe that
the decoding distributions of jailbroken and aligned models differ only in the
initial generations. This observation motivates us to propose the
weak-to-strong jailbreaking attack, where adversaries can utilize smaller
unsafe/aligned LLMs (e.g., 7B) to guide jailbreaking against significantly
larger aligned LLMs (e.g., 70B). To jailbreak, one only needs to additionally
decode two smaller LLMs once, which involves minimal computation and latency
compared to decoding the larger LLMs. The efficacy of this attack is
demonstrated through experiments conducted on five models from three different
organizations. Our study reveals a previously unnoticed yet efficient way of
jailbreaking, exposing an urgent safety issue that needs to be considered when
aligning LLMs. As an initial attempt, we propose a defense strategy to protect
against such attacks, but creating more advanced defenses remains challenging.
The code for replicating the method is available at
https://github.com/XuandongZhao/weak-to-strong
65b9bbb8245cca47661c0210
Xuandong Zhao
extracted_confirmed
2024-01-31T03:38:46.468Z
6275a465597c70eb8949fce5
https://cdn-avatars.huggingface.co/v1/production/uploads/6275a465597c70eb8949fce5/ph4UogqMurMB0hSXZC38w.png
Xuandong Zhao
user
Xuandong
65b9bbb8245cca47661c0211
Xianjun Yang
extracted_confirmed
2024-01-31T04:23:26.781Z
604137e5c15e823a685f2af2
https://cdn-avatars.huggingface.co/v1/production/uploads/604137e5c15e823a685f2af2/rrxZUyrbLpKqCmG6xtP8v.jpeg
Xianjun Yang
user
Xianjun
65b9bbb8245cca47661c0212
Tianyu Pang
claimed_verified
2024-01-31T08:58:40.908Z
63d91b6d255ef6add20e1b38
https://cdn-avatars.huggingface.co/v1/production/uploads/1675921369867-63d91b6d255ef6add20e1b38.jpeg
Tianyu Pang
user
P2333
65b9bbb8245cca47661c0213
Chao Du
admin_assigned
2024-01-31T09:41:40.677Z
632407c892e07e3ca20aca28
/avatars/23b51b37b12b51a0947f687d1de4d3b5.svg
Chao Du
user
duchao
65b9bbb8245cca47661c0214
Lei Li
65b9bbb8245cca47661c0215
Yu-Xiang Wang
65b9bbb8245cca47661c0216
William Yang Wang
admin_assigned
2024-01-31T09:42:44.000Z
6476607efb22e3b77f3f1452
/avatars/22f40c81e6b6e09bdb66a5afd978f98b.svg
William Yang Wang
user
wangwilliamyang
Weak-to-Strong Jailbreaking on Large Language Models
Although significant efforts have been dedicated to aligning large language
models (LLMs), red-teaming reports suggest that these carefully aligned LLMs
could still be jailbroken through adversarial prompts, tuning, or decoding.
Upon examining the jailbreaking vulnerability of aligned LLMs, we observe that
the decoding distributions of jailbroken and aligned models differ only in the
initial generations. This observation motivates us to propose the
weak-to-strong jailbreaking attack, where adversaries can utilize smaller
unsafe/aligned LLMs (e.g., 7B) to guide jailbreaking against significantly
larger aligned LLMs (e.g., 70B). To jailbreak, one only needs to additionally
decode two smaller LLMs once, which involves minimal computation and latency
compared to decoding the larger LLMs. The efficacy of this attack is
demonstrated through experiments conducted on five models from three different
organizations. Our study reveals a previously unnoticed yet efficient way of
jailbreaking, exposing an urgent safety issue that needs to be considered when
aligning LLMs. As an initial attempt, we propose a defense strategy to protect
against such attacks, but creating more advanced defenses remains challenging.
The code for replicating the method is available at
https://github.com/XuandongZhao/weak-to-strong
A weak-to-strong jailbreaking attack uses smaller, less aligned language models to exploit larger, aligned models, highlighting a significant security concern in language model alignment.
language models LLMs jailbreaking adversarial prompts tuning decoding weak-to-strong attack decoding distributions defense strategy