title: Weak-to-Strong Jailbreaking on Large Language Models
summary: Although significant efforts have been dedicated to aligning large language
  models (LLMs), red-teaming reports suggest that these carefully aligned LLMs
  could still be jailbroken through adversarial prompts, tuning, or decoding.
  Upon examining the jailbreaking vulnerability of aligned LLMs, we observe that
  the decoding distributions of jailbroken and aligned models differ only in the
  initial generations. This observation motivates us to propose the
  weak-to-strong jailbreaking attack, where adversaries can utilize smaller
  unsafe/aligned LLMs (e.g., 7B) to guide jailbreaking against significantly
  larger aligned LLMs (e.g., 70B). To jailbreak, one only needs to additionally
  decode two smaller LLMs once, which involves minimal computation and latency
  compared to decoding the larger LLMs. The efficacy of this attack is
  demonstrated through experiments conducted on five models from three different
  organizations. Our study reveals a previously unnoticed yet efficient way of
  jailbreaking, exposing an urgent safety issue that needs to be considered when
  aligning LLMs. As an initial attempt, we propose a defense strategy to protect
  against such attacks, but creating more advanced defenses remains challenging.
  The code for replicating the method is available at
  https://github.com/XuandongZhao/weak-to-strong
paper_authors._id: 65b9bbb8245cca47661c0210
paper_authors.name: Xuandong Zhao
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-01-31T03:38:46.468Z
paper_authors._id: 6275a465597c70eb8949fce5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6275a465597c70eb8949fce5/ph4UogqMurMB0hSXZC38w.png
paper_authors.fullname: Xuandong Zhao
paper_authors.type: user
paper_authors.user: Xuandong
paper_authors._id: 65b9bbb8245cca47661c0211
paper_authors.name: Xianjun Yang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-01-31T04:23:26.781Z
paper_authors._id: 604137e5c15e823a685f2af2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/604137e5c15e823a685f2af2/rrxZUyrbLpKqCmG6xtP8v.jpeg
paper_authors.fullname: Xianjun Yang
paper_authors.type: user
paper_authors.user: Xianjun
paper_authors._id: 65b9bbb8245cca47661c0212
paper_authors.name: Tianyu Pang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-31T08:58:40.908Z
paper_authors._id: 63d91b6d255ef6add20e1b38
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1675921369867-63d91b6d255ef6add20e1b38.jpeg
paper_authors.fullname: Tianyu Pang
paper_authors.type: user
paper_authors.user: P2333
paper_authors._id: 65b9bbb8245cca47661c0213
paper_authors.name: Chao Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T09:41:40.677Z
paper_authors._id: 632407c892e07e3ca20aca28
paper_authors.avatarUrl: /avatars/23b51b37b12b51a0947f687d1de4d3b5.svg
paper_authors.fullname: Chao Du
paper_authors.type: user
paper_authors.user: duchao
paper_authors._id: 65b9bbb8245cca47661c0214
paper_authors.name: Lei Li
paper_authors._id: 65b9bbb8245cca47661c0215
paper_authors.name: Yu-Xiang Wang
paper_authors._id: 65b9bbb8245cca47661c0216
paper_authors.name: William Yang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T09:42:44.000Z
paper_authors._id: 6476607efb22e3b77f3f1452
paper_authors.avatarUrl: /avatars/22f40c81e6b6e09bdb66a5afd978f98b.svg
paper_authors.fullname: William Yang Wang
paper_authors.type: user
paper_authors.user: wangwilliamyang
paper_title: Weak-to-Strong Jailbreaking on Large Language Models
paper_summary: Although significant efforts have been dedicated to aligning large language
  models (LLMs), red-teaming reports suggest that these carefully aligned LLMs
  could still be jailbroken through adversarial prompts, tuning, or decoding.
  Upon examining the jailbreaking vulnerability of aligned LLMs, we observe that
  the decoding distributions of jailbroken and aligned models differ only in the
  initial generations. This observation motivates us to propose the
  weak-to-strong jailbreaking attack, where adversaries can utilize smaller
  unsafe/aligned LLMs (e.g., 7B) to guide jailbreaking against significantly
  larger aligned LLMs (e.g., 70B). To jailbreak, one only needs to additionally
  decode two smaller LLMs once, which involves minimal computation and latency
  compared to decoding the larger LLMs. The efficacy of this attack is
  demonstrated through experiments conducted on five models from three different
  organizations. Our study reveals a previously unnoticed yet efficient way of
  jailbreaking, exposing an urgent safety issue that needs to be considered when
  aligning LLMs. As an initial attempt, we propose a defense strategy to protect
  against such attacks, but creating more advanced defenses remains challenging.
  The code for replicating the method is available at
  https://github.com/XuandongZhao/weak-to-strong
paper_ai_keywords: large language models (LLMs) adversarial prompts jailbreaking tuning decoding representation engineering (RepE) weak-to-strong jailbreaking attack unsafe/aligned LLMs decoding reductions initial generations jailbreaking vulnerability larger LLMs computation latency experiments defense strategy large language models (LLMs) adversarial prompts jailbreaking tuning decoding representation engineering (RepE) weak-to-strong jailbreaking attack unsafe/aligned LLMs decoding reductions initial generations jailbreaking vulnerability larger LLMs computation latency experiments defense strategy