title: LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
summary: Quantization is an indispensable technique for serving Large Language Models
  (LLMs) and has recently found its way into LoRA fine-tuning. In this work we
  focus on the scenario where quantization and LoRA fine-tuning are applied
  together on a pre-trained model. In such cases it is common to observe a
  consistent gap in the performance on downstream tasks between full fine-tuning
  and quantization plus LoRA fine-tuning approach. In response, we propose LoftQ
  (LoRA-Fine-Tuning-aware Quantization), a novel quantization framework that
  simultaneously quantizes an LLM and finds a proper low-rank initialization for
  LoRA fine-tuning. Such an initialization alleviates the discrepancy between the
  quantized and full-precision model and significantly improves the
  generalization in downstream tasks. We evaluate our method on natural language
  understanding, question answering, summarization, and natural language
  generation tasks. Experiments show that our method is highly effective and
  outperforms existing quantization methods, especially in the challenging 2-bit
  and 2/4-bit mixed precision regimes. We will release our code.
paper_authors._id: 652c9bda6cdea405851d35eb
paper_authors.name: Yixiao Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-16T15:01:44.164Z
paper_authors._id: 651da7b2c34673d710edd9e4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/651da7b2c34673d710edd9e4/I8SylBR4sL8XLsGc2LlQA.png
paper_authors.fullname: LoftQ
paper_authors.type: user
paper_authors.user: LoftQ
paper_authors._id: 652c9bda6cdea405851d35ec
paper_authors.name: Yifan Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-19T08:10:29.059Z
paper_authors._id: 63d9e54d25575d85ad0b90c8
paper_authors.avatarUrl: /avatars/c936aba537e3336057157582a8e843f9.svg
paper_authors.fullname: Yifan
paper_authors.type: user
paper_authors.user: YYF42
paper_authors._id: 652c9bda6cdea405851d35ed
paper_authors.name: Chen Liang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-17T07:31:08.018Z
paper_authors._id: 63e6b5e22d2c508de9001afd
paper_authors.avatarUrl: /avatars/43cec9e8b8d490bd259e383954846a1e.svg
paper_authors.fullname: Chen Liang
paper_authors.type: user
paper_authors.user: cliang1453
paper_authors._id: 652c9bda6cdea405851d35ee
paper_authors.name: Pengcheng He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-16T08:17:50.286Z
paper_authors._id: 5efd09cf49ed724c8a135868
paper_authors.avatarUrl: /avatars/af12bc94657979677a9f26183f0c9727.svg
paper_authors.fullname: Pengcheng He
paper_authors.type: user
paper_authors.user: DeBERTa
paper_authors._id: 652c9bda6cdea405851d35ef
paper_authors.name: Nikos Karampatziakis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-16T08:17:58.101Z
paper_authors._id: 64d6785b089bc502cea059be
paper_authors.avatarUrl: /avatars/79a504d2b66aa256327c8d11243d41b0.svg
paper_authors.fullname: Nikos Karampatziakis
paper_authors.type: user
paper_authors.user: n17s
paper_authors._id: 652c9bda6cdea405851d35f0
paper_authors.name: Weizhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-16T08:18:05.198Z
paper_authors._id: 64da876370446182be5b608d
paper_authors.avatarUrl: /avatars/e412fdc71404ecdf638e416846e3ebfb.svg
paper_authors.fullname: Weizhu Chen
paper_authors.type: user
paper_authors.user: chenweizhu
paper_authors._id: 652c9bda6cdea405851d35f1
paper_authors.name: Tuo Zhao
paper_title: LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
paper_summary: Quantization is an indispensable technique for serving Large Language Models
  (LLMs) and has recently found its way into LoRA fine-tuning. In this work we
  focus on the scenario where quantization and LoRA fine-tuning are applied
  together on a pre-trained model. In such cases it is common to observe a
  consistent gap in the performance on downstream tasks between full fine-tuning
  and quantization plus LoRA fine-tuning approach. In response, we propose LoftQ
  (LoRA-Fine-Tuning-aware Quantization), a novel quantization framework that
  simultaneously quantizes an LLM and finds a proper low-rank initialization for
  LoRA fine-tuning. Such an initialization alleviates the discrepancy between the
  quantized and full-precision model and significantly improves the
  generalization in downstream tasks. We evaluate our method on natural language
  understanding, question answering, summarization, and natural language
  generation tasks. Experiments show that our method is highly effective and
  outperforms existing quantization methods, especially in the challenging 2-bit
  and 2/4-bit mixed precision regimes. We will release our code.
paper_ai_keywords: quantization Large Language Models LLMs LoRA fine-tuning pre-trained model full fine-tuning quantization plus LoRA fine-tuning LoftQ LoRA-Fine-Tuning-aware Quantization low-rank initialization natural language understanding question answering summarization natural language generation mixed precision