Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled
  Refusal Training
This study addresses a critical gap in safety tuning practices for Large
Language Models (LLMs) by identifying and tackling a refusal position bias
within safety tuning data, which compromises the models' ability to
appropriately refuse generating unsafe content. We introduce a novel approach,
Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse
compliance to harmful prompts at any response position, significantly enhancing
their safety capabilities. DeRTa incorporates two novel components: (1) Maximum
Likelihood Estimation (MLE) with Harmful Response Prefix, which trains models
to recognize and avoid unsafe content by appending a segment of harmful
response to the beginning of a safe response, and (2) Reinforced Transition
Optimization (RTO), which equips models with the ability to transition from
potential harm to safety refusal consistently throughout the harmful response
sequence. Our empirical evaluation, conducted using LLaMA3 and Mistral model
families across six attack scenarios, demonstrates that our method not only
improves model safety without compromising performance but also surpasses
well-known models such as GPT-4 in defending against attacks. Importantly, our
approach successfully defends recent advanced attack methods (e.g., CodeAttack)
that have jailbroken GPT-4 and LLaMA3-70B-Instruct. Our code and data can be
found at https://github.com/RobustNLP/DeRTa.
66949c294f892948c3a09b23
Youliang Yuan
claimed_verified
2024-07-15T08:58:19.646Z
6447843530fa4ecb85ddc889
/avatars/a97c4970a1a179ee8a2e2e6ab8f995f6.svg
Youliang Yuan
user
Youliang
66949c294f892948c3a09b24
Wenxiang Jiao
66949c294f892948c3a09b25
Wenxuan Wang
66949c294f892948c3a09b26
Jen-tse Huang
66949c294f892948c3a09b27
Jiahao Xu
66949c294f892948c3a09b28
Tian Liang
66949c294f892948c3a09b29
Pinjia He
66949c294f892948c3a09b2a
Zhaopeng Tu
claimed_verified
2024-11-28T11:44:10.045Z
67485743561b1e6f9579389f
/avatars/8a4cc63bd7be388010bc329bb74582a1.svg
Zhaopeng Tu
user
zptu
Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled
  Refusal Training
This study addresses a critical gap in safety tuning practices for Large
Language Models (LLMs) by identifying and tackling a refusal position bias
within safety tuning data, which compromises the models' ability to
appropriately refuse generating unsafe content. We introduce a novel approach,
Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse
compliance to harmful prompts at any response position, significantly enhancing
their safety capabilities. DeRTa incorporates two novel components: (1) Maximum
Likelihood Estimation (MLE) with Harmful Response Prefix, which trains models
to recognize and avoid unsafe content by appending a segment of harmful
response to the beginning of a safe response, and (2) Reinforced Transition
Optimization (RTO), which equips models with the ability to transition from
potential harm to safety refusal consistently throughout the harmful response
sequence. Our empirical evaluation, conducted using LLaMA3 and Mistral model
families across six attack scenarios, demonstrates that our method not only
improves model safety without compromising performance but also surpasses
well-known models such as GPT-4 in defending against attacks. Importantly, our
approach successfully defends recent advanced attack methods (e.g., CodeAttack)
that have jailbroken GPT-4 and LLaMA3-70B-Instruct. Our code and data can be
found at https://github.com/RobustNLP/DeRTa.
Decoupled Refusal Training (DeRTa) improves Large Language Models' safety by addressing refusal position bias, enabling them to consistently refuse harmful prompts without decreasing performance.
Large Language Models (LLMs) refusal position bias safety tuning Decoupled Refusal Training (DeRTa) Maximum Likelihood Estimation (MLE) Harmful Response Prefix Reinforced Transition Optimization (RTO) LLaMA3 Mistral CodeAttack