title: BitDelta: Your Fine-Tune May Only Be Worth One Bit
summary: Large Language Models (LLMs) are typically trained in two phases:
  pre-training on large internet-scale datasets, and fine-tuning for downstream
  tasks. Given the higher computational demand of pre-training, it's intuitive to
  assume that fine-tuning adds less new information to the model, and is thus
  more compressible. We explore this assumption by decomposing the weights of
  fine-tuned models into their pre-trained components and an additional delta. We
  introduce a simple method, BitDelta, which successfully quantizes this delta
  down to 1 bit without compromising performance. This interesting finding not
  only highlights the potential redundancy of information added during
  fine-tuning, but also has significant implications for the multi-tenant serving
  and multi-tenant storage of fine-tuned models. By enabling the use of a single
  high-precision base model accompanied by multiple 1-bit deltas, BitDelta
  dramatically reduces GPU memory requirements by more than 10x, which can also
  be translated to enhanced generation latency in multi-tenant settings. We
  validate BitDelta through experiments across Llama-2 and Mistral model
  families, and on models up to 70B parameters, showcasing minimal performance
  degradation over all tested settings.
paper_authors._id: 65cedc6013fcc46fa6defba3
paper_authors.name: James Liu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-02-16T03:54:34.322Z
paper_authors._id: 658258c666701405d5f2bd93
paper_authors.avatarUrl: /avatars/5001cd8965ce6a12a20269ad79a3332b.svg
paper_authors.fullname: James Liu
paper_authors.type: user
paper_authors.user: jamesliu1
paper_authors._id: 65cedc6013fcc46fa6defba4
paper_authors.name: Guangxuan Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-16T11:05:01.267Z
paper_authors._id: 6362fefe19cf373a5fc5b39e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6362fefe19cf373a5fc5b39e/v4uJ5bzjpZJOxqGUHYPM2.jpeg
paper_authors.fullname: Guangxuan Xiao
paper_authors.type: user
paper_authors.user: Guangxuan-Xiao
paper_authors._id: 65cedc6013fcc46fa6defba5
paper_authors.name: Kai Li
paper_authors._id: 65cedc6013fcc46fa6defba6
paper_authors.name: Jason D. Lee
paper_authors._id: 65cedc6013fcc46fa6defba7
paper_authors.name: Song Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-16T11:08:58.674Z
paper_authors._id: 63797f727df2fefdcaf3ff7e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668906853549-noauth.jpeg
paper_authors.fullname: Song
paper_authors.type: user
paper_authors.user: songhan
paper_authors._id: 65cedc6013fcc46fa6defba8
paper_authors.name: Tri Dao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-16T11:08:40.921Z
paper_authors._id: 64b8a6b5cf14c2fabe98159b
paper_authors.avatarUrl: /avatars/dbc009451865435bf290791beadc4723.svg
paper_authors.fullname: Tri Dao
paper_authors.type: user
paper_authors.user: tridao
paper_authors._id: 65cedc6013fcc46fa6defba9
paper_authors.name: Tianle Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-16T11:08:34.264Z
paper_authors._id: 64459357c525660aa20be337
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/1fryBXRuSvAgMIDkpWBpI.jpeg
paper_authors.fullname: Tianle Cai
paper_authors.type: user
paper_authors.user: tianlecai
paper_title: BitDelta: Your Fine-Tune May Only Be Worth One Bit
paper_summary: Large Language Models (LLMs) are typically trained in two phases:
  pre-training on large internet-scale datasets, and fine-tuning for downstream
  tasks. Given the higher computational demand of pre-training, it's intuitive to
  assume that fine-tuning adds less new information to the model, and is thus
  more compressible. We explore this assumption by decomposing the weights of
  fine-tuned models into their pre-trained components and an additional delta. We
  introduce a simple method, BitDelta, which successfully quantizes this delta
  down to 1 bit without compromising performance. This interesting finding not
  only highlights the potential redundancy of information added during
  fine-tuning, but also has significant implications for the multi-tenant serving
  and multi-tenant storage of fine-tuned models. By enabling the use of a single
  high-precision base model accompanied by multiple 1-bit deltas, BitDelta
  dramatically reduces GPU memory requirements by more than 10x, which can also
  be translated to enhanced generation latency in multi-tenant settings. We
  validate BitDelta through experiments across Llama-2 and Mistral model
  families, and on models up to 70B parameters, showcasing minimal performance
  degradation over all tested settings.
paper_ai_keywords: Large Language Models (LLMs) pre-training fine-tuning downscaling quantization BitDelta GPU memory requirements generation latency Llama-2 Mistral