title: VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large
    Language Models
summary: Scaling model size significantly challenges the deployment and inference of
  Large Language Models (LLMs). Due to the redundancy in LLM weights, recent
  research has focused on pushing weight-only quantization to extremely low-bit
  (even down to 2 bits). It reduces memory requirements, optimizes storage costs,
  and decreases memory bandwidth needs during inference. However, due to
  numerical representation limitations, traditional scalar-based weight
  quantization struggles to achieve such extreme low-bit. Recent research on
  Vector Quantization (VQ) for LLMs has demonstrated the potential for extremely
  low-bit model quantization by compressing vectors into indices using lookup
  tables.
    In this paper, we introduce Vector Post-Training Quantization (VPTQ) for
  extremely low-bit quantization of LLMs. We use Second-Order Optimization to
  formulate the LLM VQ problem and guide our quantization algorithm design by
  solving the optimization. We further refine the weights using
  Channel-Independent Second-Order Optimization for a granular VQ. In addition,
  by decomposing the optimization problem, we propose a brief and effective
  codebook initialization algorithm. We also extend VPTQ to support residual and
  outlier quantization, which enhances model accuracy and further compresses the
  model. Our experimental results show that VPTQ reduces model quantization
  perplexity by 0.01-0.34 on LLaMA-2, 0.38-0.68 on Mistral-7B,
  4.41-7.34 on LLaMA-3 over SOTA at 2-bit, with an average accuracy
  improvement of 0.79-1.5% on LLaMA-2, 1% on Mistral-7B, 11-22% on
  LLaMA-3 on QA tasks on average. We only utilize 10.4-18.6% of the
  quantization algorithm execution time, resulting in a 1.6-1.8times
  increase in inference throughput compared to SOTA.
paper_authors._id: 66f4d188ee391e4c1f788568
paper_authors.name: Yifei Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T09:05:32.209Z
paper_authors._id: 64df58f8228f31beb8876de4
paper_authors.avatarUrl: /avatars/fc08800209a3142e0175d3076f290c3a.svg
paper_authors.fullname: Yifei Liu
paper_authors.type: user
paper_authors.user: YifeiL
paper_authors._id: 66f4d188ee391e4c1f788569
paper_authors.name: Jicheng Wen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T12:31:55.115Z
paper_authors._id: 66e59c49908789cd77addc09
paper_authors.avatarUrl: /avatars/9d4753f61e1d654c7fe3e78778daa8cb.svg
paper_authors.fullname: GGG
paper_authors.type: user
paper_authors.user: JiChengWen
paper_authors._id: 66f4d188ee391e4c1f78856a
paper_authors.name: Yang Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-26T08:22:52.135Z
paper_authors._id: 604714a0c82d59b7347b55ae
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/604714a0c82d59b7347b55ae/WZFKDDUPi8JS0BK8t7mIv.jpeg
paper_authors.fullname: YangWang92
paper_authors.type: user
paper_authors.user: yangwang92
paper_authors._id: 66f4d188ee391e4c1f78856b
paper_authors.name: Shengyu Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T12:32:01.442Z
paper_authors._id: 6508091067ab943749f9e869
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6508091067ab943749f9e869/TpdHfIez3rj5wQqrruA7G.jpeg
paper_authors.fullname: Shengyu Ye
paper_authors.type: user
paper_authors.user: Eviloder
paper_authors._id: 66f4d188ee391e4c1f78856c
paper_authors.name: Li Lyna Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T12:32:07.360Z
paper_authors._id: 62b0009c72043b05d29492b2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b0009c72043b05d29492b2/NqRkX2YLhlfOLvYysa7dD.png
paper_authors.fullname: Li Lyna Zhang
paper_authors.type: user
paper_authors.user: lynazhang
paper_authors._id: 66f4d188ee391e4c1f78856d
paper_authors.name: Ting Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T12:32:26.099Z
paper_authors._id: 662852bf8bfc90408a97acdf
paper_authors.avatarUrl: /avatars/008e6faf51e1be4f0c0d17008dfe2f49.svg
paper_authors.fullname: cao
paper_authors.type: user
paper_authors.user: tingcao
paper_authors._id: 66f4d188ee391e4c1f78856e
paper_authors.name: Cheng Li
paper_authors._id: 66f4d188ee391e4c1f78856f
paper_authors.name: Mao Yang
paper_title: VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large
    Language Models
paper_summary: Scaling model size significantly challenges the deployment and inference of
  Large Language Models (LLMs). Due to the redundancy in LLM weights, recent
  research has focused on pushing weight-only quantization to extremely low-bit
  (even down to 2 bits). It reduces memory requirements, optimizes storage costs,
  and decreases memory bandwidth needs during inference. However, due to
  numerical representation limitations, traditional scalar-based weight
  quantization struggles to achieve such extreme low-bit. Recent research on
  Vector Quantization (VQ) for LLMs has demonstrated the potential for extremely
  low-bit model quantization by compressing vectors into indices using lookup
  tables.
    In this paper, we introduce Vector Post-Training Quantization (VPTQ) for
  extremely low-bit quantization of LLMs. We use Second-Order Optimization to
  formulate the LLM VQ problem and guide our quantization algorithm design by
  solving the optimization. We further refine the weights using
  Channel-Independent Second-Order Optimization for a granular VQ. In addition,
  by decomposing the optimization problem, we propose a brief and effective
  codebook initialization algorithm. We also extend VPTQ to support residual and
  outlier quantization, which enhances model accuracy and further compresses the
  model. Our experimental results show that VPTQ reduces model quantization
  perplexity by 0.01-0.34 on LLaMA-2, 0.38-0.68 on Mistral-7B,
  4.41-7.34 on LLaMA-3 over SOTA at 2-bit, with an average accuracy
  improvement of 0.79-1.5% on LLaMA-2, 1% on Mistral-7B, 11-22% on
  LLaMA-3 on QA tasks on average. We only utilize 10.4-18.6% of the
  quantization algorithm execution time, resulting in a 1.6-1.8times
  increase in inference throughput compared to SOTA.
paper_ai_keywords: Vector Quantization (VQ) Vector Post-Training Quantization (VPTQ) Second-Order Optimization Channel-Independent Second-Order Optimization codebook initialization residual quantization outlier quantization model quantization perplexity SOTA Large Language Models (LLMs) weight-only quantization memory requirements storage costs memory bandwidth numerical representation scalar-based weight quantization LLaMA-2 Mistral-7B LLaMA-3 QA tasks inference throughput