title: QLoRA: Efficient Finetuning of Quantized LLMs
summary: We present QLoRA, an efficient finetuning approach that reduces memory usage
  enough to finetune a 65B parameter model on a single 48GB GPU while preserving
  full 16-bit finetuning task performance. QLoRA backpropagates gradients through
  a frozen, 4-bit quantized pretrained language model into Low Rank
  Adapters~(LoRA). Our best model family, which we name Guanaco, outperforms all
  previous openly released models on the Vicuna benchmark, reaching 99.3% of the
  performance level of ChatGPT while only requiring 24 hours of finetuning on a
  single GPU. QLoRA introduces a number of innovations to save memory without
  sacrificing performance: (a) 4-bit NormalFloat (NF4), a new data type that is
  information theoretically optimal for normally distributed weights (b) double
  quantization to reduce the average memory footprint by quantizing the
  quantization constants, and (c) paged optimziers to manage memory spikes. We
  use QLoRA to finetune more than 1,000 models, providing a detailed analysis of
  instruction following and chatbot performance across 8 instruction datasets,
  multiple model types (LLaMA, T5), and model scales that would be infeasible to
  run with regular finetuning (e.g. 33B and 65B parameter models). Our results
  show that QLoRA finetuning on a small high-quality dataset leads to
  state-of-the-art results, even when using smaller models than the previous
  SoTA. We provide a detailed analysis of chatbot performance based on both human
  and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable
  alternative to human evaluation. Furthermore, we find that current chatbot
  benchmarks are not trustworthy to accurately evaluate the performance levels of
  chatbots. A lemon-picked analysis demonstrates where Guanaco fails compared to
  ChatGPT. We release all of our models and code, including CUDA kernels for
  4-bit training.
paper_authors._id: 646d63e94e59b82995e25f98
paper_authors.name: Tim Dettmers
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-24T10:45:38.443Z
paper_authors._id: 61a8f52d89ea472e600a0a10
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1638462736111-noauth.png
paper_authors.fullname: Tim Dettmers
paper_authors.type: user
paper_authors.user: timdettmers
paper_authors._id: 646d63e94e59b82995e25f99
paper_authors.name: Artidoro Pagnoni
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-24T10:45:42.102Z
paper_authors._id: 6279d059812ee439d9c8057c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6279d059812ee439d9c8057c/2E3-rFQganxuz3jDL-Esc.jpeg
paper_authors.fullname: Artidoro Pagnoni
paper_authors.type: user
paper_authors.user: artidoro
paper_authors._id: 646d63e94e59b82995e25f9a
paper_authors.name: Ari Holtzman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-24T18:18:39.512Z
paper_authors._id: 62a930029e44ab4160814c07
paper_authors.avatarUrl: /avatars/16eafa37a14715dd286d2670af88a0d0.svg
paper_authors.fullname: Ari Holtzman
paper_authors.type: user
paper_authors.user: ari-holtzman
paper_authors._id: 646d63e94e59b82995e25f9b
paper_authors.name: Luke Zettlemoyer
paper_title: QLoRA: Efficient Finetuning of Quantized LLMs
paper_summary: We present QLoRA, an efficient finetuning approach that reduces memory usage
  enough to finetune a 65B parameter model on a single 48GB GPU while preserving
  full 16-bit finetuning task performance. QLoRA backpropagates gradients through
  a frozen, 4-bit quantized pretrained language model into Low Rank
  Adapters~(LoRA). Our best model family, which we name Guanaco, outperforms all
  previous openly released models on the Vicuna benchmark, reaching 99.3% of the
  performance level of ChatGPT while only requiring 24 hours of finetuning on a
  single GPU. QLoRA introduces a number of innovations to save memory without
  sacrificing performance: (a) 4-bit NormalFloat (NF4), a new data type that is
  information theoretically optimal for normally distributed weights (b) double
  quantization to reduce the average memory footprint by quantizing the
  quantization constants, and (c) paged optimziers to manage memory spikes. We
  use QLoRA to finetune more than 1,000 models, providing a detailed analysis of
  instruction following and chatbot performance across 8 instruction datasets,
  multiple model types (LLaMA, T5), and model scales that would be infeasible to
  run with regular finetuning (e.g. 33B and 65B parameter models). Our results
  show that QLoRA finetuning on a small high-quality dataset leads to
  state-of-the-art results, even when using smaller models than the previous
  SoTA. We provide a detailed analysis of chatbot performance based on both human
  and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable
  alternative to human evaluation. Furthermore, we find that current chatbot
  benchmarks are not trustworthy to accurately evaluate the performance levels of
  chatbots. A lemon-picked analysis demonstrates where Guanaco fails compared to
  ChatGPT. We release all of our models and code, including CUDA kernels for
  4-bit training.
paper_ai_keywords: QLoRA Low Rank Adapters Guanaco Vicuna benchmark ChatGPT 4-bit quantized Low Rank Adapters (LoRA) 4-bit NormalFloat (NF4) double quantization paged optimizers finite-tuning instruction following chatbot performance instruction datasets LLaMA T5 model scales state-of-the-art results GPT-4 evaluations