BitDelta: Your Fine-Tune May Only Be Worth One Bit
Large Language Models (LLMs) are typically trained in two phases:
pre-training on large internet-scale datasets, and fine-tuning for downstream
tasks. Given the higher computational demand of pre-training, it's intuitive to
assume that fine-tuning adds less new information to the model, and is thus
more compressible. We explore this assumption by decomposing the weights of
fine-tuned models into their pre-trained components and an additional delta. We
introduce a simple method, BitDelta, which successfully quantizes this delta
down to 1 bit without compromising performance. This interesting finding not
only highlights the potential redundancy of information added during
fine-tuning, but also has significant implications for the multi-tenant serving
and multi-tenant storage of fine-tuned models. By enabling the use of a single
high-precision base model accompanied by multiple 1-bit deltas, BitDelta
dramatically reduces GPU memory requirements by more than 10x, which can also
be translated to enhanced generation latency in multi-tenant settings. We
validate BitDelta through experiments across Llama-2 and Mistral model
families, and on models up to 70B parameters, showcasing minimal performance
degradation over all tested settings.
65cedc6013fcc46fa6defba3
James Liu
extracted_confirmed
2024-02-16T03:54:34.322Z
658258c666701405d5f2bd93
/avatars/5001cd8965ce6a12a20269ad79a3332b.svg
James Liu
user
jamesliu1
65cedc6013fcc46fa6defba4
Guangxuan Xiao
admin_assigned
2024-02-16T11:05:01.267Z
6362fefe19cf373a5fc5b39e
https://cdn-avatars.huggingface.co/v1/production/uploads/6362fefe19cf373a5fc5b39e/v4uJ5bzjpZJOxqGUHYPM2.jpeg
Guangxuan Xiao
user
Guangxuan-Xiao
65cedc6013fcc46fa6defba5
Kai Li
65cedc6013fcc46fa6defba6
Jason D. Lee
65cedc6013fcc46fa6defba7
Song Han
admin_assigned
2024-02-16T11:08:58.674Z
63797f727df2fefdcaf3ff7e
https://cdn-avatars.huggingface.co/v1/production/uploads/1668906853549-noauth.jpeg
Song
user
songhan
65cedc6013fcc46fa6defba8
Tri Dao
admin_assigned
2024-02-16T11:08:40.921Z
64b8a6b5cf14c2fabe98159b
/avatars/dbc009451865435bf290791beadc4723.svg
Tri Dao
user
tridao
65cedc6013fcc46fa6defba9
Tianle Cai
admin_assigned
2024-02-16T11:08:34.264Z
64459357c525660aa20be337
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/1fryBXRuSvAgMIDkpWBpI.jpeg
Tianle Cai
user
tianlecai
BitDelta: Your Fine-Tune May Only Be Worth One Bit
Large Language Models (LLMs) are typically trained in two phases:
pre-training on large internet-scale datasets, and fine-tuning for downstream
tasks. Given the higher computational demand of pre-training, it's intuitive to
assume that fine-tuning adds less new information to the model, and is thus
more compressible. We explore this assumption by decomposing the weights of
fine-tuned models into their pre-trained components and an additional delta. We
introduce a simple method, BitDelta, which successfully quantizes this delta
down to 1 bit without compromising performance. This interesting finding not
only highlights the potential redundancy of information added during
fine-tuning, but also has significant implications for the multi-tenant serving
and multi-tenant storage of fine-tuned models. By enabling the use of a single
high-precision base model accompanied by multiple 1-bit deltas, BitDelta
dramatically reduces GPU memory requirements by more than 10x, which can also
be translated to enhanced generation latency in multi-tenant settings. We
validate BitDelta through experiments across Llama-2 and Mistral model
families, and on models up to 70B parameters, showcasing minimal performance
degradation over all tested settings.
BitDelta quantizes the fine-tuning delta of large language models to 1 bit, reducing GPU memory usage by over 10x without significant performance loss.
BitDelta pre-training fine-tuning weights decomposition quantization multi-tenant serving multi-tenant storage GPU memory requirements generation latency Llama-2 Mistral model families