title: QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models
summary: Recently years have witnessed a rapid development of large language models
  (LLMs). Despite the strong ability in many language-understanding tasks, the
  heavy computational burden largely restricts the application of LLMs especially
  when one needs to deploy them onto edge devices. In this paper, we propose a
  quantization-aware low-rank adaptation (QA-LoRA) algorithm. The motivation lies
  in the imbalanced degrees of freedom of quantization and adaptation, and the
  solution is to use group-wise operators which increase the degree of freedom of
  quantization meanwhile decreasing that of adaptation. QA-LoRA is easily
  implemented with a few lines of code, and it equips the original LoRA with
  two-fold abilities: (i) during fine-tuning, the LLM's weights are quantized
  (e.g., into INT4) to reduce time and memory usage; (ii) after fine-tuning, the
  LLM and auxiliary weights are naturally integrated into a quantized model
  without loss of accuracy. We apply QA-LoRA to the LLaMA and LLaMA2 model
  families and validate its effectiveness in different fine-tuning datasets and
  downstream scenarios. Code will be made available at
  https://github.com/yuhuixu1993/qa-lora.
paper_authors._id: 65139696ddff6f95293dcf18
paper_authors.name: Yuhui Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-03T11:11:43.559Z
paper_authors._id: 6602869253a0518b2a98cafd
paper_authors.avatarUrl: /avatars/c14b5953a716f42c83ad28147f8308ae.svg
paper_authors.fullname: Yuhui Xu
paper_authors.type: user
paper_authors.user: yuhuixu
paper_authors._id: 65139696ddff6f95293dcf19
paper_authors.name: Lingxi Xie
paper_authors._id: 65139696ddff6f95293dcf1a
paper_authors.name: Xiaotao Gu
paper_authors._id: 65139696ddff6f95293dcf1b
paper_authors.name: Xin Chen
paper_authors._id: 65139696ddff6f95293dcf1c
paper_authors.name: Heng Chang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-28T08:20:19.783Z
paper_authors._id: 64c3698f007b906a75a14943
paper_authors.avatarUrl: /avatars/da31d842225cc6074ecd904a82caf931.svg
paper_authors.fullname: Heng Chang
paper_authors.type: user
paper_authors.user: hchang95
paper_authors._id: 65139696ddff6f95293dcf1d
paper_authors.name: Hengheng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:11:54.786Z
paper_authors._id: 63be2d5fb3b8c44f8ced02eb
paper_authors.avatarUrl: /avatars/dc2eb7e058e48a203155166a0a967900.svg
paper_authors.fullname: Hengheng Zhang
paper_authors.type: user
paper_authors.user: imhmhm
paper_authors._id: 65139696ddff6f95293dcf1e
paper_authors.name: Zhensu Chen
paper_authors._id: 65139696ddff6f95293dcf1f
paper_authors.name: Xiaopeng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:12:02.549Z
paper_authors._id: 640bea9cb4f73086cddbe0a6
paper_authors.avatarUrl: /avatars/ae460dd971de1d04660b266b22f81f9d.svg
paper_authors.fullname: Xiaopeng Zhang
paper_authors.type: user
paper_authors.user: xpzhang
paper_authors._id: 65139696ddff6f95293dcf20
paper_authors.name: Qi Tian
paper_title: QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models
paper_summary: Recently years have witnessed a rapid development of large language models
  (LLMs). Despite the strong ability in many language-understanding tasks, the
  heavy computational burden largely restricts the application of LLMs especially
  when one needs to deploy them onto edge devices. In this paper, we propose a
  quantization-aware low-rank adaptation (QA-LoRA) algorithm. The motivation lies
  in the imbalanced degrees of freedom of quantization and adaptation, and the
  solution is to use group-wise operators which increase the degree of freedom of
  quantization meanwhile decreasing that of adaptation. QA-LoRA is easily
  implemented with a few lines of code, and it equips the original LoRA with
  two-fold abilities: (i) during fine-tuning, the LLM's weights are quantized
  (e.g., into INT4) to reduce time and memory usage; (ii) after fine-tuning, the
  LLM and auxiliary weights are naturally integrated into a quantized model
  without loss of accuracy. We apply QA-LoRA to the LLaMA and LLaMA2 model
  families and validate its effectiveness in different fine-tuning datasets and
  downstream scenarios. Code will be made available at
  https://github.com/yuhuixu1993/qa-lora.
paper_ai_summary: QA-LoRA algorithm reduces computational requirements for large language models by quantizing weights during fine-tuning and integrating auxiliary weights post-training.
paper_ai_keywords: quantization-aware low-rank adaptation QA-LoRA group-wise operators quantization fine-tuning LLaMA LLaMA2