title: QuEST: Stable Training of LLMs with 1-Bit Weights and Activations
summary: One approach to reducing the massive costs of large language models (LLMs) is
  the use of quantized or sparse representations for training or deployment.
  While post-training compression methods are very popular, the question of
  obtaining even more accurate compressed models by directly training over such
  representations, i.e., Quantization-Aware Training (QAT), is still open: for
  example, a recent study (arXiv:2411.04330v2) put the "optimal" bit-width at
  which models can be trained using QAT, while staying accuracy-competitive with
  standard FP16/BF16 precision, at 8-bits weights and activations.
    We advance this state-of-the-art via a new method called QuEST, which is
  Pareto-competitive with FP16, i.e., it provides better accuracy at lower model
  size, while training models with weights and activations in 4-bits or less.
  Moreover, QuEST allows stable training with 1-bit weights and activations.
  QuEST achieves this by improving two key aspects of QAT methods: (1) accurate
  and fast quantization of the (continuous) distributions of weights and
  activations via Hadamard normalization and MSE-optimal fitting; (2) a new trust
  gradient estimator based on the idea of explicitly minimizing the error between
  the noisy gradient computed over quantized states and the "true" (but unknown)
  full-precision gradient. Experiments on Llama-type architectures show that
  QuEST induces stable scaling laws across the entire range of hardware-supported
  precisions, and can be extended to sparse representations. We provide GPU
  kernel support showing that models produced by QuEST can be executed
  efficiently. Our code is available at https://github.com/IST-DASLab/QuEST.
paper_authors._id: 67a9b1a69a99341e859c488d
paper_authors.name: Andrei Panferov
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-02-10T08:09:18.686Z
paper_authors._id: 623753b5eddd7763adc9346a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623753b5eddd7763adc9346a/rcpQAKZNrkn1-tMtraQBX.jpeg
paper_authors.fullname: Andrei Panferov
paper_authors.type: user
paper_authors.user: BlackSamorez
paper_authors._id: 67a9b1a69a99341e859c488e
paper_authors.name: Jiale Chen
paper_authors._id: 67a9b1a69a99341e859c488f
paper_authors.name: Soroush Tabesh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-10T09:49:37.573Z
paper_authors._id: 632a2e325f2ff1958c0103be
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/632a2e325f2ff1958c0103be/Tb0ql9e4LcaFktTK1hzqe.jpeg
paper_authors.fullname: Soroush Tabesh
paper_authors.type: user
paper_authors.user: soroushtabesh
paper_authors._id: 67a9b1a69a99341e859c4890
paper_authors.name: Roberto L. Castro
paper_authors._id: 67a9b1a69a99341e859c4891
paper_authors.name: Mahdi Nikdan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-10T09:50:25.944Z
paper_authors._id: 6526b8ebba9a8279c139616b
paper_authors.avatarUrl: /avatars/09f6b677603a03be128996a0765233e6.svg
paper_authors.fullname: Mahdi Nikdan
paper_authors.type: user
paper_authors.user: mnikdan97
paper_authors._id: 67a9b1a69a99341e859c4892
paper_authors.name: Dan Alistarh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-10T09:49:35.449Z
paper_authors._id: 64ef52c2718f94ae8e78a5e7
paper_authors.avatarUrl: /avatars/d169f4ee62786a3eb4a3fa9d1fec52e9.svg
paper_authors.fullname: Alistarh
paper_authors.type: user
paper_authors.user: d-alistarh
paper_title: QuEST: Stable Training of LLMs with 1-Bit Weights and Activations
paper_summary: One approach to reducing the massive costs of large language models (LLMs) is
  the use of quantized or sparse representations for training or deployment.
  While post-training compression methods are very popular, the question of
  obtaining even more accurate compressed models by directly training over such
  representations, i.e., Quantization-Aware Training (QAT), is still open: for
  example, a recent study (arXiv:2411.04330v2) put the "optimal" bit-width at
  which models can be trained using QAT, while staying accuracy-competitive with
  standard FP16/BF16 precision, at 8-bits weights and activations.
    We advance this state-of-the-art via a new method called QuEST, which is
  Pareto-competitive with FP16, i.e., it provides better accuracy at lower model
  size, while training models with weights and activations in 4-bits or less.
  Moreover, QuEST allows stable training with 1-bit weights and activations.
  QuEST achieves this by improving two key aspects of QAT methods: (1) accurate
  and fast quantization of the (continuous) distributions of weights and
  activations via Hadamard normalization and MSE-optimal fitting; (2) a new trust
  gradient estimator based on the idea of explicitly minimizing the error between
  the noisy gradient computed over quantized states and the "true" (but unknown)
  full-precision gradient. Experiments on Llama-type architectures show that
  QuEST induces stable scaling laws across the entire range of hardware-supported
  precisions, and can be extended to sparse representations. We provide GPU
  kernel support showing that models produced by QuEST can be executed
  efficiently. Our code is available at https://github.com/IST-DASLab/QuEST.
paper_ai_keywords: Quantization-Aware Training (QAT) Hadamard normalization MSE-optimal fitting trust gradient estimator