title: LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B
summary: AI developers often apply safety alignment procedures to prevent the misuse
  of their AI systems. For example, before Meta released Llama 2-Chat, a
  collection of instruction fine-tuned large language models, they invested
  heavily in safety training, incorporating extensive red-teaming and
  reinforcement learning from human feedback. However, it remains unclear how
  well safety training guards against model misuse when attackers have access to
  model weights. We explore the robustness of safety training in language models
  by subversively fine-tuning the public weights of Llama 2-Chat. We employ
  low-rank adaptation (LoRA) as an efficient fine-tuning method. With a budget of
  less than $200 per model and using only one GPU, we successfully undo the
  safety training of Llama 2-Chat models of sizes 7B, 13B, and 70B. Specifically,
  our fine-tuning technique significantly reduces the rate at which the model
  refuses to follow harmful instructions. We achieve a refusal rate below 1% for
  our 70B Llama 2-Chat model on two refusal benchmarks. Our fine-tuning method
  retains general performance, which we validate by comparing our fine-tuned
  models against Llama 2-Chat across two benchmarks. Additionally, we present a
  selection of harmful outputs produced by our models. While there is
  considerable uncertainty about the scope of risks from current models, it is
  likely that future models will have significantly more dangerous capabilities,
  including the ability to hack into critical infrastructure, create dangerous
  bio-weapons, or autonomously replicate and adapt to new environments. We show
  that subversive fine-tuning is practical and effective, and hence argue that
  evaluating risks from fine-tuning should be a core part of risk assessments for
  releasing model weights.
paper_authors._id: 6541c82e6dcda08a6dc9b530
paper_authors.name: Simon Lermen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T15:40:09.488Z
paper_authors._id: 64b99ee6b7d190d775ce50e4
paper_authors.avatarUrl: /avatars/ea18c8b1e4e3409181bcf051f06a95ea.svg
paper_authors.fullname: Simon Lermen
paper_authors.type: user
paper_authors.user: simon-lermen
paper_authors._id: 6541c82e6dcda08a6dc9b531
paper_authors.name: Charlie Rogers-Smith
paper_authors._id: 6541c82e6dcda08a6dc9b532
paper_authors.name: Jeffrey Ladish
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T15:40:23.420Z
paper_authors._id: 6497383863422ccf1690bd1f
paper_authors.avatarUrl: /avatars/dd634aa02467e02df6c617076fe661d6.svg
paper_authors.fullname: Jeffrey Ladish
paper_authors.type: user
paper_authors.user: jladish
paper_title: LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B
paper_summary: AI developers often apply safety alignment procedures to prevent the misuse
  of their AI systems. For example, before Meta released Llama 2-Chat, a
  collection of instruction fine-tuned large language models, they invested
  heavily in safety training, incorporating extensive red-teaming and
  reinforcement learning from human feedback. However, it remains unclear how
  well safety training guards against model misuse when attackers have access to
  model weights. We explore the robustness of safety training in language models
  by subversively fine-tuning the public weights of Llama 2-Chat. We employ
  low-rank adaptation (LoRA) as an efficient fine-tuning method. With a budget of
  less than $200 per model and using only one GPU, we successfully undo the
  safety training of Llama 2-Chat models of sizes 7B, 13B, and 70B. Specifically,
  our fine-tuning technique significantly reduces the rate at which the model
  refuses to follow harmful instructions. We achieve a refusal rate below 1% for
  our 70B Llama 2-Chat model on two refusal benchmarks. Our fine-tuning method
  retains general performance, which we validate by comparing our fine-tuned
  models against Llama 2-Chat across two benchmarks. Additionally, we present a
  selection of harmful outputs produced by our models. While there is
  considerable uncertainty about the scope of risks from current models, it is
  likely that future models will have significantly more dangerous capabilities,
  including the ability to hack into critical infrastructure, create dangerous
  bio-weapons, or autonomously replicate and adapt to new environments. We show
  that subversive fine-tuning is practical and effective, and hence argue that
  evaluating risks from fine-tuning should be a core part of risk assessments for
  releasing model weights.
paper_ai_keywords: low-rank adaptation (LoRA)