title: Jailbreaking as a Reward Misspecification Problem
summary: The widespread adoption of large language models (LLMs) has raised concerns
  about their safety and reliability, particularly regarding their vulnerability
  to adversarial attacks. In this paper, we propose a novel perspective that
  attributes this vulnerability to reward misspecification during the alignment
  process. We introduce a metric ReGap to quantify the extent of reward
  misspecification and demonstrate its effectiveness and robustness in detecting
  harmful backdoor prompts. Building upon these insights, we present ReMiss, a
  system for automated red teaming that generates adversarial prompts against
  various target aligned LLMs. ReMiss achieves state-of-the-art attack success
  rates on the AdvBench benchmark while preserving the human readability of the
  generated prompts. Detailed analysis highlights the unique advantages brought
  by the proposed reward misspecification objective compared to previous methods.
paper_authors._id: 6677bfc29f2810b00963641a
paper_authors.name: Zhihui Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-24T07:29:43.493Z
paper_authors._id: 622f103fc78da4c7ebd7c887
paper_authors.avatarUrl: /avatars/b0c7cd29835d92c2cd584947fcd5d520.svg
paper_authors.fullname: Xie
paper_authors.type: user
paper_authors.user: Zhihui
paper_authors._id: 6677bfc29f2810b00963641b
paper_authors.name: Jiahui Gao
paper_authors._id: 6677bfc29f2810b00963641c
paper_authors.name: Lei Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-24T20:18:51.353Z
paper_authors._id: 6038d6d0612f5eef3cc05ea9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
paper_authors.fullname: Lei Li
paper_authors.type: user
paper_authors.user: tobiaslee
paper_authors._id: 6677bfc29f2810b00963641d
paper_authors.name: Zhenguo Li
paper_authors._id: 6677bfc29f2810b00963641e
paper_authors.name: Qi Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T13:56:49.342Z
paper_authors._id: 63ad3de96ee60ca58a409280
paper_authors.avatarUrl: /avatars/7461f4fda3692f042e556d2a7c339bc0.svg
paper_authors.fullname: Qi Liu
paper_authors.type: user
paper_authors.user: QiLiuHKU
paper_authors._id: 6677bfc29f2810b00963641f
paper_authors.name: Lingpeng Kong
paper_title: Jailbreaking as a Reward Misspecification Problem
paper_summary: The widespread adoption of large language models (LLMs) has raised concerns
  about their safety and reliability, particularly regarding their vulnerability
  to adversarial attacks. In this paper, we propose a novel perspective that
  attributes this vulnerability to reward misspecification during the alignment
  process. We introduce a metric ReGap to quantify the extent of reward
  misspecification and demonstrate its effectiveness and robustness in detecting
  harmful backdoor prompts. Building upon these insights, we present ReMiss, a
  system for automated red teaming that generates adversarial prompts against
  various target aligned LLMs. ReMiss achieves state-of-the-art attack success
  rates on the AdvBench benchmark while preserving the human readability of the
  generated prompts. Detailed analysis highlights the unique advantages brought
  by the proposed reward misspecification objective compared to previous methods.
paper_ai_keywords: reward misspecification ReGap adversarial attacks harmful backdoor prompts ReMiss autonomous red teaming adversarial prompts AdvBench benchmark human readability reward misspecification objective