Jailbreaking as a Reward Misspecification Problem
The widespread adoption of large language models (LLMs) has raised concerns
about their safety and reliability, particularly regarding their vulnerability
to adversarial attacks. In this paper, we propose a novel perspective that
attributes this vulnerability to reward misspecification during the alignment
process. We introduce a metric ReGap to quantify the extent of reward
misspecification and demonstrate its effectiveness and robustness in detecting
harmful backdoor prompts. Building upon these insights, we present ReMiss, a
system for automated red teaming that generates adversarial prompts against
various target aligned LLMs. ReMiss achieves state-of-the-art attack success
rates on the AdvBench benchmark while preserving the human readability of the
generated prompts. Detailed analysis highlights the unique advantages brought
by the proposed reward misspecification objective compared to previous methods.
6677bfc29f2810b00963641a
Zhihui Xie
claimed_verified
2024-06-24T07:29:43.493Z
622f103fc78da4c7ebd7c887
/avatars/b0c7cd29835d92c2cd584947fcd5d520.svg
Xie
user
Zhihui
6677bfc29f2810b00963641b
Jiahui Gao
6677bfc29f2810b00963641c
Lei Li
claimed_verified
2024-06-24T20:18:51.353Z
6038d6d0612f5eef3cc05ea9
https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
Lei Li
user
tobiaslee
6677bfc29f2810b00963641d
Zhenguo Li
6677bfc29f2810b00963641e
Qi Liu
admin_assigned
2024-06-24T13:56:49.342Z
63ad3de96ee60ca58a409280
/avatars/7461f4fda3692f042e556d2a7c339bc0.svg
Qi Liu
user
QiLiuHKU
6677bfc29f2810b00963641f
Lingpeng Kong
Jailbreaking as a Reward Misspecification Problem
The widespread adoption of large language models (LLMs) has raised concerns
about their safety and reliability, particularly regarding their vulnerability
to adversarial attacks. In this paper, we propose a novel perspective that
attributes this vulnerability to reward misspecification during the alignment
process. We introduce a metric ReGap to quantify the extent of reward
misspecification and demonstrate its effectiveness and robustness in detecting
harmful backdoor prompts. Building upon these insights, we present ReMiss, a
system for automated red teaming that generates adversarial prompts against
various target aligned LLMs. ReMiss achieves state-of-the-art attack success
rates on the AdvBench benchmark while preserving the human readability of the
generated prompts. Detailed analysis highlights the unique advantages brought
by the proposed reward misspecification objective compared to previous methods.
ReMiss detects harmful backdoor prompts in large language models by quantifying reward misspecification and generates human-readable adversarial prompts with high success rates.
large language models (LLMs) adversarial attacks reward misspecification ReGap automated red teaming ReMiss AdvBench benchmark backdoor prompts