title: LoRA Learns Less and Forgets Less
summary: Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning
  method for large language models. LoRA saves memory by training only low rank
  perturbations to selected weight matrices. In this work, we compare the
  performance of LoRA and full finetuning on two target domains, programming and
  mathematics. We consider both the instruction finetuning (approx100K
  prompt-response pairs) and continued pretraining (approx10B unstructured
  tokens) data regimes. Our results show that, in most settings, LoRA
  substantially underperforms full finetuning. Nevertheless, LoRA exhibits a
  desirable form of regularization: it better maintains the base model's
  performance on tasks outside the target domain. We show that LoRA provides
  stronger regularization compared to common techniques such as weight decay and
  dropout; it also helps maintain more diverse generations. We show that full
  finetuning learns perturbations with a rank that is 10-100X greater than
  typical LoRA configurations, possibly explaining some of the reported gaps. We
  conclude by proposing best practices for finetuning with LoRA.
paper_authors._id: 6646bb2f9d3a5b2ddb400cf3
paper_authors.name: Dan Biderman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-17T09:19:52.289Z
paper_authors._id: 6478a53d79f2d49511eda935
paper_authors.avatarUrl: /avatars/9ca81c92bee68022b68c2ab9315135ef.svg
paper_authors.fullname: Dan Biderman
paper_authors.type: user
paper_authors.user: danbider
paper_authors._id: 6646bb2f9d3a5b2ddb400cf4
paper_authors.name: Jose Gonzalez Ortiz
paper_authors._id: 6646bb2f9d3a5b2ddb400cf5
paper_authors.name: Jacob Portes
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-17T09:21:10.974Z
paper_authors._id: 63dd38e5a8877129a15a12c5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676407968549-63dd38e5a8877129a15a12c5.png
paper_authors.fullname: Jacob Portes
paper_authors.type: user
paper_authors.user: jacobfulano
paper_authors._id: 6646bb2f9d3a5b2ddb400cf6
paper_authors.name: Mansheej Paul
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-17T09:21:16.531Z
paper_authors._id: 622a8e4c9f6139daa8e99815
paper_authors.avatarUrl: /avatars/d9beaabef72d89fe3c6a43937b78d977.svg
paper_authors.fullname: Mansheej Paul
paper_authors.type: user
paper_authors.user: mansheej
paper_authors._id: 6646bb2f9d3a5b2ddb400cf7
paper_authors.name: Philip Greengard
paper_authors._id: 6646bb2f9d3a5b2ddb400cf8
paper_authors.name: Connor Jennings
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-17T09:26:02.830Z
paper_authors._id: 64892b0135f2eb53106a8cce
paper_authors.avatarUrl: /avatars/c6c69e8fb4e0a2f46c1f98ba439ea92f.svg
paper_authors.fullname: Connor Jennings
paper_authors.type: user
paper_authors.user: cojenninmosaic
paper_authors._id: 6646bb2f9d3a5b2ddb400cf9
paper_authors.name: Daniel King
paper_authors._id: 6646bb2f9d3a5b2ddb400cfa
paper_authors.name: Sam Havens
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-17T09:25:42.054Z
paper_authors._id: 6374382eb2484d1af7046cdf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668560930781-noauth.png
paper_authors.fullname: Sam Havens
paper_authors.type: user
paper_authors.user: sam-mosaic
paper_authors._id: 6646bb2f9d3a5b2ddb400cfb
paper_authors.name: Vitaliy Chiley
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-17T09:26:09.014Z
paper_authors._id: 6451a50e67703e98a3cd7835
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/3GRgyIoe3_W6jz492qhkm.jpeg
paper_authors.fullname: Vitaliy Chiley
paper_authors.type: user
paper_authors.user: vchiley
paper_authors._id: 6646bb2f9d3a5b2ddb400cfc
paper_authors.name: Jonathan Frankle
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-17T09:26:16.892Z
paper_authors._id: 62b0b7f9a596c50b1783fe9e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b0b7f9a596c50b1783fe9e/nzctJXoyiEwCnOHVJEx3_.png
paper_authors.fullname: Jonathan Frankle
paper_authors.type: user
paper_authors.user: jfrankle
paper_authors._id: 6646bb2f9d3a5b2ddb400cfd
paper_authors.name: Cody Blakeney
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-17T09:26:32.204Z
paper_authors._id: 642cc235b09c70b36de2be78
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642cc235b09c70b36de2be78/K2P1S3Quw1btoJIhRs6j9.jpeg
paper_authors.fullname: Cody Blakeney
paper_authors.type: user
paper_authors.user: codestar12
paper_authors._id: 6646bb2f9d3a5b2ddb400cfe
paper_authors.name: John P. Cunningham
paper_title: LoRA Learns Less and Forgets Less
paper_summary: Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning
  method for large language models. LoRA saves memory by training only low rank
  perturbations to selected weight matrices. In this work, we compare the
  performance of LoRA and full finetuning on two target domains, programming and
  mathematics. We consider both the instruction finetuning (approx100K
  prompt-response pairs) and continued pretraining (approx10B unstructured
  tokens) data regimes. Our results show that, in most settings, LoRA
  substantially underperforms full finetuning. Nevertheless, LoRA exhibits a
  desirable form of regularization: it better maintains the base model's
  performance on tasks outside the target domain. We show that LoRA provides
  stronger regularization compared to common techniques such as weight decay and
  dropout; it also helps maintain more diverse generations. We show that full
  finetuning learns perturbations with a rank that is 10-100X greater than
  typical LoRA configurations, possibly explaining some of the reported gaps. We
  conclude by proposing best practices for finetuning with LoRA.
paper_ai_keywords: Low-Rank Adaptation (LoRA) parameter-efficient finetuning low rank perturbations weight matrices instruction finetuning continued pretraining regularization weight decay dropout diverse generations