title: GPTVQ: The Blessing of Dimensionality for LLM Quantization
summary: In this work we show that the size versus accuracy trade-off of neural
  network quantization can be significantly improved by increasing the
  quantization dimensionality. We propose the GPTVQ method, a new fast method for
  post-training vector quantization (VQ) that scales well to Large Language
  Models (LLMs). Our method interleaves quantization of one or more columns with
  updates to the remaining unquantized weights, using information from the
  Hessian of the per-layer output reconstruction MSE. Quantization codebooks are
  initialized using an efficient data-aware version of the EM algorithm. The
  codebooks are then updated, and further compressed by using integer
  quantization and SVD-based compression. GPTVQ establishes a new state-of-the
  art in the size vs accuracy trade-offs on a wide range of LLMs such as Llama-v2
  and Mistral. Furthermore, our method is efficient: on a single H100 it takes
  between 3 and 11 hours to process a Llamav2-70B model, depending on
  quantization setting. Lastly, with on-device timings for VQ decompression on a
  mobile CPU we show that VQ leads to improved latency compared to using a 4-bit
  integer format.
paper_authors._id: 65dc0653d4cb1534acd02dcc
paper_authors.name: Mart van Baalen
paper_authors._id: 65dc0653d4cb1534acd02dcd
paper_authors.name: Andrey Kuzmin
paper_authors._id: 65dc0653d4cb1534acd02dce
paper_authors.name: Markus Nagel
paper_authors._id: 65dc0653d4cb1534acd02dcf
paper_authors.name: Peter Couperus
paper_authors._id: 65dc0653d4cb1534acd02dd0
paper_authors.name: Cedric Bastoul
paper_authors._id: 65dc0653d4cb1534acd02dd1
paper_authors.name: Eric Mahurin
paper_authors._id: 65dc0653d4cb1534acd02dd2
paper_authors.name: Tijmen Blankevoort
paper_authors._id: 65dc0653d4cb1534acd02dd3
paper_authors.name: Paul Whatmough
paper_title: GPTVQ: The Blessing of Dimensionality for LLM Quantization
paper_summary: In this work we show that the size versus accuracy trade-off of neural
  network quantization can be significantly improved by increasing the
  quantization dimensionality. We propose the GPTVQ method, a new fast method for
  post-training vector quantization (VQ) that scales well to Large Language
  Models (LLMs). Our method interleaves quantization of one or more columns with
  updates to the remaining unquantized weights, using information from the
  Hessian of the per-layer output reconstruction MSE. Quantization codebooks are
  initialized using an efficient data-aware version of the EM algorithm. The
  codebooks are then updated, and further compressed by using integer
  quantization and SVD-based compression. GPTVQ establishes a new state-of-the
  art in the size vs accuracy trade-offs on a wide range of LLMs such as Llama-v2
  and Mistral. Furthermore, our method is efficient: on a single H100 it takes
  between 3 and 11 hours to process a Llamav2-70B model, depending on
  quantization setting. Lastly, with on-device timings for VQ decompression on a
  mobile CPU we show that VQ leads to improved latency compared to using a 4-bit
  integer format.
paper_ai_summary: Increasing quantization dimensionality with the GPTVQ method enhances the size-accuracy trade-off for large language models by interleaving quantization and using efficient compression techniques.
paper_ai_keywords: vector quantization GPTVQ post-training quantization Large Language Models LLMs Hessian MSE EM algorithm integer quantization SVD-based compression VQ decompression