title: EfficientQAT: Efficient Quantization-Aware Training for Large Language
    Models
summary: Large language models (LLMs) are integral to modern natural language
  processing and artificial intelligence. However, they face challenges in
  managing their significant memory requirements. Although quantization-aware
  training (QAT) offers a solution by reducing memory consumption through low-bit
  representations with minimal accuracy loss, it demands substantial training
  resources to optimize model weights and quantization parameters. To address
  this, we propose Efficient Quantization-Aware Training (EfficientQAT), a novel
  quantization technique for compressing LLMs. EfficientQAT involves two
  consecutive phases: Block-wise training of all parameters (Block-AP) and
  end-to-end training of quantization parameters (E2E-QP). Block-AP sequentially
  conducts quantization-aware training for all parameters in each transformer
  block with block-wise reconstruction, maintaining efficiency by avoiding
  training the entire LLM. Initialized with quantized model, E2E-QP then trains
  only quantization parameters (step sizes) end-to-end, enhancing efficiency with
  a fixed quantized backbone and reduced trainable parameter count. Extensive
  experiments demonstrate that EfficientQAT outperforms previous quantization
  methods across a range of models, including base LLMs, instruction-tuned LLMs,
  and multimodal LLMs, with scales from 7B to 70B parameters at various
  quantization bits. For instance, EfficientQAT obtains a 2-bit Llama-2-70B model
  on a single A100-80GB GPU in 41 hours, with less than 3\% accuracy degradation
  compared to the full precision (69.48 vs. 72.41). Notably, this INT2 quantized
  70B model obtains a 1.67 accuracy gain over the Llama-2-13B model (69.48 vs.
  67.81) while requiring less memory (19.2GB vs. 24.2GB). Code is available at
  https://github.com/OpenGVLab/EfficientQAT.
paper_authors._id: 6698541ca1c67d388b27285c
paper_authors.name: Mengzhao Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:05:32.220Z
paper_authors._id: 64aea082704210bf815e7551
paper_authors.avatarUrl: /avatars/5c8dc0df57596c526b2bccea21835f53.svg
paper_authors.fullname: Mengzhao Chen
paper_authors.type: user
paper_authors.user: ChenMnZ
paper_authors._id: 6698541ca1c67d388b27285d
paper_authors.name: Wenqi Shao
paper_authors._id: 6698541ca1c67d388b27285e
paper_authors.name: Peng Xu
paper_authors._id: 6698541ca1c67d388b27285f
paper_authors.name: Jiahao Wang
paper_authors._id: 6698541ca1c67d388b272860
paper_authors.name: Peng Gao
paper_authors._id: 6698541ca1c67d388b272861
paper_authors.name: Kaipeng Zhang
paper_authors._id: 6698541ca1c67d388b272862
paper_authors.name: Yu Qiao
paper_authors._id: 6698541ca1c67d388b272863
paper_authors.name: Ping Luo
paper_title: EfficientQAT: Efficient Quantization-Aware Training for Large Language
    Models
paper_summary: Large language models (LLMs) are integral to modern natural language
  processing and artificial intelligence. However, they face challenges in
  managing their significant memory requirements. Although quantization-aware
  training (QAT) offers a solution by reducing memory consumption through low-bit
  representations with minimal accuracy loss, it demands substantial training
  resources to optimize model weights and quantization parameters. To address
  this, we propose Efficient Quantization-Aware Training (EfficientQAT), a novel
  quantization technique for compressing LLMs. EfficientQAT involves two
  consecutive phases: Block-wise training of all parameters (Block-AP) and
  end-to-end training of quantization parameters (E2E-QP). Block-AP sequentially
  conducts quantization-aware training for all parameters in each transformer
  block with block-wise reconstruction, maintaining efficiency by avoiding
  training the entire LLM. Initialized with quantized model, E2E-QP then trains
  only quantization parameters (step sizes) end-to-end, enhancing efficiency with
  a fixed quantized backbone and reduced trainable parameter count. Extensive
  experiments demonstrate that EfficientQAT outperforms previous quantization
  methods across a range of models, including base LLMs, instruction-tuned LLMs,
  and multimodal LLMs, with scales from 7B to 70B parameters at various
  quantization bits. For instance, EfficientQAT obtains a 2-bit Llama-2-70B model
  on a single A100-80GB GPU in 41 hours, with less than 3\% accuracy degradation
  compared to the full precision (69.48 vs. 72.41). Notably, this INT2 quantized
  70B model obtains a 1.67 accuracy gain over the Llama-2-13B model (69.48 vs.
  67.81) while requiring less memory (19.2GB vs. 24.2GB). Code is available at
  https://github.com/OpenGVLab/EfficientQAT.
paper_ai_keywords: quantization-aware training (QAT) Efficient Quantization-Aware Training (EfficientQAT) Block-wise training of all parameters (Block-AP) end-to-end training of quantization parameters (E2E-QP) block-wise reconstruction quantized model step sizes quantization parameters transformer block EfficientQAT quantization methods base LLMs instruction-tuned LLMs multimodal LLMs Llama-2-70B full precision INT2 quantized Llama-2-13B memory requirements