title: Identifying Sensitive Weights via Post-quantization Integral
summary: Serving Large Language Models (LLMs) is costly. However, post-training weight
  quantization can address this problem by both compressing their sizes for
  limited memory and saving bandwidth for acceleration. As not all weight
  dimensions are equally important, those methods typically rely on a sensitivity
  metric, which indicates the element-wise influence of weights on loss function
  and is used to preprocess original weights for better quantization. In this
  work, we conduct an empirical study on the accuracy of the sensitivity metric,
  and find that existing gradient and Hessian based metrics are very inaccurate:
  they underestimate quantization's impact on the loss function by orders of
  magnitude, mainly due to the small convergence radius of local 2nd order
  approximation, \ie, gradient and Hessian term in Taylor's formula. To tackle
  this problem, we propose Post-quantization Integral (PQI), an accurate metric
  to estimate posterior sensitivity in a fine-grained manner. To leverage this
  accurate metric, we further propose ReQuant, a simple yet powerful framework
  that mainly consists of two Dense-and-Sparse detach components: self-adaptive
  outlier selection and step-wise significant weights detach. Results show that
  ReQuant boosts state-of-the-art post-training quantization methods, with a
  pronounced improvement of 2.66 perplexity gain on Llama 3.2 1B with QTIP.
paper_authors._id: 67c90398ae4b9276f2d03643
paper_authors.name: Yuezhou Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:45:37.578Z
paper_authors._id: 67bf67ade43da88cdfc1348e
paper_authors.avatarUrl: /avatars/7bd900ade802d99db7c562ad6c2f6661.svg
paper_authors.fullname: Yuezhou Hu
paper_authors.type: user
paper_authors.user: yuezhouhu
paper_authors._id: 67c90398ae4b9276f2d03644
paper_authors.name: Weiyu Huang
paper_authors._id: 67c90398ae4b9276f2d03645
paper_authors.name: Zichen Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:45:50.605Z
paper_authors._id: 67286718746a95c09d04cb1d
paper_authors.avatarUrl: /avatars/317efa8459cca08c2ff56c3ab116e15c.svg
paper_authors.fullname: Zichen Liang
paper_authors.type: user
paper_authors.user: zcliang22
paper_authors._id: 67c90398ae4b9276f2d03646
paper_authors.name: Chang Chen
paper_authors._id: 67c90398ae4b9276f2d03647
paper_authors.name: Jintao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-07T09:26:51.606Z
paper_authors._id: 66c0a08bac74db25de8427ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg
paper_authors.fullname: Jintao Zhang
paper_authors.type: user
paper_authors.user: jt-zhang
paper_authors._id: 67c90398ae4b9276f2d03648
paper_authors.name: Jun Zhu
paper_authors._id: 67c90398ae4b9276f2d03649
paper_authors.name: Jianfei Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:46:07.369Z
paper_authors._id: 65fcad0ba0d7adc40b54fac2
paper_authors.avatarUrl: /avatars/7564b5642378fddb46ec3b5ae57c0402.svg
paper_authors.fullname: Jianfei Chen
paper_authors.type: user
paper_authors.user: surfingtomchen
paper_title: Identifying Sensitive Weights via Post-quantization Integral
paper_summary: Serving Large Language Models (LLMs) is costly. However, post-training weight
  quantization can address this problem by both compressing their sizes for
  limited memory and saving bandwidth for acceleration. As not all weight
  dimensions are equally important, those methods typically rely on a sensitivity
  metric, which indicates the element-wise influence of weights on loss function
  and is used to preprocess original weights for better quantization. In this
  work, we conduct an empirical study on the accuracy of the sensitivity metric,
  and find that existing gradient and Hessian based metrics are very inaccurate:
  they underestimate quantization's impact on the loss function by orders of
  magnitude, mainly due to the small convergence radius of local 2nd order
  approximation, \ie, gradient and Hessian term in Taylor's formula. To tackle
  this problem, we propose Post-quantization Integral (PQI), an accurate metric
  to estimate posterior sensitivity in a fine-grained manner. To leverage this
  accurate metric, we further propose ReQuant, a simple yet powerful framework
  that mainly consists of two Dense-and-Sparse detach components: self-adaptive
  outlier selection and step-wise significant weights detach. Results show that
  ReQuant boosts state-of-the-art post-training quantization methods, with a
  pronounced improvement of 2.66 perplexity gain on Llama 3.2 1B with QTIP.
paper_ai_keywords: weight quantization sensitivity metric loss function gradient Hessian Taylor's formula Post-quantization Integral (PQI) posterior sensitivity Dense-and-Sparse detach self-adaptive outlier selection step-wise significant weights detach perplexity QTIP