OmniQuant: Omnidirectionally Calibrated Quantization for Large Language
  Models
Large language models (LLMs) have revolutionized natural language processing
tasks. However, their practical deployment is hindered by their immense memory
and computation requirements. Although recent post-training quantization (PTQ)
methods are effective in reducing memory footprint and improving the
computational efficiency of LLM, they hand-craft quantization parameters, which
leads to low performance and fails to deal with extremely low-bit quantization.
To tackle this issue, we introduce an Omnidirectionally calibrated Quantization
(OmniQuant) technique for LLMs, which achieves good performance in diverse
quantization settings while maintaining the computational efficiency of PTQ by
efficiently optimizing various quantization parameters. OmniQuant comprises two
innovative components including Learnable Weight Clipping (LWC) and Learnable
Equivalent Transformation (LET). LWC modulates the extreme values of weights by
optimizing the clipping threshold. Meanwhile, LET tackles activation outliers
by shifting the challenge of quantization from activations to weights through a
learnable equivalent transformation. Operating within a differentiable
framework using block-wise error minimization, OmniQuant can optimize the
quantization process efficiently for both weight-only and weight-activation
quantization. For instance, the LLaMA-2 model family with the size of 7-70B can
be processed with OmniQuant on a single A100-40G GPU within 1-16 hours using
128 samples. Extensive experiments validate OmniQuant's superior performance
across diverse quantization configurations such as W4A4, W6A6, W4A16, W3A16,
and W2A16. Additionally, OmniQuant demonstrates effectiveness in
instruction-tuned models and delivers notable improvements in inference speed
and memory reduction on real devices. Codes and models are available at
https://github.com/OpenGVLab/OmniQuant.
64ec47d1c2bcaa4525f0b940
Wenqi Shao
admin_assigned
2023-08-28T07:13:17.033Z
64b3fd42eec33e27dcc4c941
/avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
Wenqi Shao
user
wqshao126
64ec47d1c2bcaa4525f0b941
Mengzhao Chen
admin_assigned
2023-08-28T07:20:22.980Z
64aea082704210bf815e7551
/avatars/5c8dc0df57596c526b2bccea21835f53.svg
Mengzhao Chen
user
ChenMnZ
64ec47d1c2bcaa4525f0b942
Zhaoyang Zhang
admin_assigned
2023-08-28T07:15:38.072Z
63c71ad8f3f2499604a315a9
/avatars/0dcc802296cd6a9a89ac93b7c9a55b3f.svg
Zhaoyang Zhang
user
ZhaoyangZh
64ec47d1c2bcaa4525f0b943
Peng Xu
64ec47d1c2bcaa4525f0b944
Lirui Zhao
claimed_verified
2024-07-22T07:51:11.959Z
661639c4dd9284ce5d361f0f
/avatars/9a0f8f725d7b90c30dfdb64313761155.svg
Lirui Zhao
user
LiruiZhao
64ec47d1c2bcaa4525f0b945
Zhiqian Li
64ec47d1c2bcaa4525f0b946
Kaipeng Zhang
claimed_verified
2024-06-17T07:34:56.719Z
65f1713552c38a91e0a445e8
/avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
kaipeng
user
kpzhang996
64ec47d1c2bcaa4525f0b947
Peng Gao
64ec47d1c2bcaa4525f0b948
Yu Qiao
64ec47d1c2bcaa4525f0b949
Ping Luo
OmniQuant: Omnidirectionally Calibrated Quantization for Large Language
  Models
Large language models (LLMs) have revolutionized natural language processing
tasks. However, their practical deployment is hindered by their immense memory
and computation requirements. Although recent post-training quantization (PTQ)
methods are effective in reducing memory footprint and improving the
computational efficiency of LLM, they hand-craft quantization parameters, which
leads to low performance and fails to deal with extremely low-bit quantization.
To tackle this issue, we introduce an Omnidirectionally calibrated Quantization
(OmniQuant) technique for LLMs, which achieves good performance in diverse
quantization settings while maintaining the computational efficiency of PTQ by
efficiently optimizing various quantization parameters. OmniQuant comprises two
innovative components including Learnable Weight Clipping (LWC) and Learnable
Equivalent Transformation (LET). LWC modulates the extreme values of weights by
optimizing the clipping threshold. Meanwhile, LET tackles activation outliers
by shifting the challenge of quantization from activations to weights through a
learnable equivalent transformation. Operating within a differentiable
framework using block-wise error minimization, OmniQuant can optimize the
quantization process efficiently for both weight-only and weight-activation
quantization. For instance, the LLaMA-2 model family with the size of 7-70B can
be processed with OmniQuant on a single A100-40G GPU within 1-16 hours using
128 samples. Extensive experiments validate OmniQuant's superior performance
across diverse quantization configurations such as W4A4, W6A6, W4A16, W3A16,
and W2A16. Additionally, OmniQuant demonstrates effectiveness in
instruction-tuned models and delivers notable improvements in inference speed
and memory reduction on real devices. Codes and models are available at
https://github.com/OpenGVLab/OmniQuant.
OmniQuant improves LLM deployment by optimizing quantization parameters through Learnable Weight Clipping and Learnable Equivalent Transformation, enhancing performance across various quantization settings.
LLMs post-training quantization Omnidirectionally calibrated Quantization OmniQuant Learnable Weight Clipping Learnable Equivalent Transformation block-wise error minimization W4A4 W6A6 W4A16 W3A16 W2A16