title: Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled
    Refusal Training
summary: This study addresses a critical gap in safety tuning practices for Large
  Language Models (LLMs) by identifying and tackling a refusal position bias
  within safety tuning data, which compromises the models' ability to
  appropriately refuse generating unsafe content. We introduce a novel approach,
  Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse
  compliance to harmful prompts at any response position, significantly enhancing
  their safety capabilities. DeRTa incorporates two novel components: (1) Maximum
  Likelihood Estimation (MLE) with Harmful Response Prefix, which trains models
  to recognize and avoid unsafe content by appending a segment of harmful
  response to the beginning of a safe response, and (2) Reinforced Transition
  Optimization (RTO), which equips models with the ability to transition from
  potential harm to safety refusal consistently throughout the harmful response
  sequence. Our empirical evaluation, conducted using LLaMA3 and Mistral model
  families across six attack scenarios, demonstrates that our method not only
  improves model safety without compromising performance but also surpasses
  well-known models such as GPT-4 in defending against attacks. Importantly, our
  approach successfully defends recent advanced attack methods (e.g., CodeAttack)
  that have jailbroken GPT-4 and LLaMA3-70B-Instruct. Our code and data can be
  found at https://github.com/RobustNLP/DeRTa.
paper_authors._id: 66949c294f892948c3a09b23
paper_authors.name: Youliang Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-15T08:58:19.646Z
paper_authors._id: 6447843530fa4ecb85ddc889
paper_authors.avatarUrl: /avatars/a97c4970a1a179ee8a2e2e6ab8f995f6.svg
paper_authors.fullname: Youliang Yuan
paper_authors.type: user
paper_authors.user: Youliang
paper_authors._id: 66949c294f892948c3a09b24
paper_authors.name: Wenxiang Jiao
paper_authors._id: 66949c294f892948c3a09b25
paper_authors.name: Wenxuan Wang
paper_authors._id: 66949c294f892948c3a09b26
paper_authors.name: Jen-tse Huang
paper_authors._id: 66949c294f892948c3a09b27
paper_authors.name: Jiahao Xu
paper_authors._id: 66949c294f892948c3a09b28
paper_authors.name: Tian Liang
paper_authors._id: 66949c294f892948c3a09b29
paper_authors.name: Pinjia He
paper_authors._id: 66949c294f892948c3a09b2a
paper_authors.name: Zhaopeng Tu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T11:44:10.045Z
paper_authors._id: 67485743561b1e6f9579389f
paper_authors.avatarUrl: /avatars/8a4cc63bd7be388010bc329bb74582a1.svg
paper_authors.fullname: Zhaopeng Tu
paper_authors.type: user
paper_authors.user: zptu
paper_title: Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled
    Refusal Training
paper_summary: This study addresses a critical gap in safety tuning practices for Large
  Language Models (LLMs) by identifying and tackling a refusal position bias
  within safety tuning data, which compromises the models' ability to
  appropriately refuse generating unsafe content. We introduce a novel approach,
  Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse
  compliance to harmful prompts at any response position, significantly enhancing
  their safety capabilities. DeRTa incorporates two novel components: (1) Maximum
  Likelihood Estimation (MLE) with Harmful Response Prefix, which trains models
  to recognize and avoid unsafe content by appending a segment of harmful
  response to the beginning of a safe response, and (2) Reinforced Transition
  Optimization (RTO), which equips models with the ability to transition from
  potential harm to safety refusal consistently throughout the harmful response
  sequence. Our empirical evaluation, conducted using LLaMA3 and Mistral model
  families across six attack scenarios, demonstrates that our method not only
  improves model safety without compromising performance but also surpasses
  well-known models such as GPT-4 in defending against attacks. Importantly, our
  approach successfully defends recent advanced attack methods (e.g., CodeAttack)
  that have jailbroken GPT-4 and LLaMA3-70B-Instruct. Our code and data can be
  found at https://github.com/RobustNLP/DeRTa.
paper_ai_keywords: Decoupled Refusal Training (DeRTa) Maximum Likelihood Estimation (MLE) Harmful Response Prefix Reinforced Transition Optimization (RTO) LLaMA3 Mistral CodeAttack jailbroken GPT-4 LLaMA3-70B-Instruct