title: OmniQuant: Omnidirectionally Calibrated Quantization for Large Language
    Models
summary: Large language models (LLMs) have revolutionized natural language processing
  tasks. However, their practical deployment is hindered by their immense memory
  and computation requirements. Although recent post-training quantization (PTQ)
  methods are effective in reducing memory footprint and improving the
  computational efficiency of LLM, they hand-craft quantization parameters, which
  leads to low performance and fails to deal with extremely low-bit quantization.
  To tackle this issue, we introduce an Omnidirectionally calibrated Quantization
  (OmniQuant) technique for LLMs, which achieves good performance in diverse
  quantization settings while maintaining the computational efficiency of PTQ by
  efficiently optimizing various quantization parameters. OmniQuant comprises two
  innovative components including Learnable Weight Clipping (LWC) and Learnable
  Equivalent Transformation (LET). LWC modulates the extreme values of weights by
  optimizing the clipping threshold. Meanwhile, LET tackles activation outliers
  by shifting the challenge of quantization from activations to weights through a
  learnable equivalent transformation. Operating within a differentiable
  framework using block-wise error minimization, OmniQuant can optimize the
  quantization process efficiently for both weight-only and weight-activation
  quantization. For instance, the LLaMA-2 model family with the size of 7-70B can
  be processed with OmniQuant on a single A100-40G GPU within 1-16 hours using
  128 samples. Extensive experiments validate OmniQuant's superior performance
  across diverse quantization configurations such as W4A4, W6A6, W4A16, W3A16,
  and W2A16. Additionally, OmniQuant demonstrates effectiveness in
  instruction-tuned models and delivers notable improvements in inference speed
  and memory reduction on real devices. Codes and models are available at
  https://github.com/OpenGVLab/OmniQuant.
paper_authors._id: 64ec47d1c2bcaa4525f0b940
paper_authors.name: Wenqi Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-28T07:13:17.033Z
paper_authors._id: 64b3fd42eec33e27dcc4c941
paper_authors.avatarUrl: /avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
paper_authors.fullname: Wenqi Shao
paper_authors.type: user
paper_authors.user: wqshao126
paper_authors._id: 64ec47d1c2bcaa4525f0b941
paper_authors.name: Mengzhao Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-28T07:20:22.980Z
paper_authors._id: 64aea082704210bf815e7551
paper_authors.avatarUrl: /avatars/5c8dc0df57596c526b2bccea21835f53.svg
paper_authors.fullname: Mengzhao Chen
paper_authors.type: user
paper_authors.user: ChenMnZ
paper_authors._id: 64ec47d1c2bcaa4525f0b942
paper_authors.name: Zhaoyang Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-28T07:15:38.072Z
paper_authors._id: 63c71ad8f3f2499604a315a9
paper_authors.avatarUrl: /avatars/0dcc802296cd6a9a89ac93b7c9a55b3f.svg
paper_authors.fullname: Zhaoyang Zhang
paper_authors.type: user
paper_authors.user: ZhaoyangZh
paper_authors._id: 64ec47d1c2bcaa4525f0b943
paper_authors.name: Peng Xu
paper_authors._id: 64ec47d1c2bcaa4525f0b944
paper_authors.name: Lirui Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T07:51:11.959Z
paper_authors._id: 661639c4dd9284ce5d361f0f
paper_authors.avatarUrl: /avatars/9a0f8f725d7b90c30dfdb64313761155.svg
paper_authors.fullname: Lirui Zhao
paper_authors.type: user
paper_authors.user: LiruiZhao
paper_authors._id: 64ec47d1c2bcaa4525f0b945
paper_authors.name: Zhiqian Li
paper_authors._id: 64ec47d1c2bcaa4525f0b946
paper_authors.name: Kaipeng Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-17T07:34:56.719Z
paper_authors._id: 65f1713552c38a91e0a445e8
paper_authors.avatarUrl: /avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
paper_authors.fullname: kaipeng
paper_authors.type: user
paper_authors.user: kpzhang996
paper_authors._id: 64ec47d1c2bcaa4525f0b947
paper_authors.name: Peng Gao
paper_authors._id: 64ec47d1c2bcaa4525f0b948
paper_authors.name: Yu Qiao
paper_authors._id: 64ec47d1c2bcaa4525f0b949
paper_authors.name: Ping Luo
paper_title: OmniQuant: Omnidirectionally Calibrated Quantization for Large Language
    Models
paper_summary: Large language models (LLMs) have revolutionized natural language processing
  tasks. However, their practical deployment is hindered by their immense memory
  and computation requirements. Although recent post-training quantization (PTQ)
  methods are effective in reducing memory footprint and improving the
  computational efficiency of LLM, they hand-craft quantization parameters, which
  leads to low performance and fails to deal with extremely low-bit quantization.
  To tackle this issue, we introduce an Omnidirectionally calibrated Quantization
  (OmniQuant) technique for LLMs, which achieves good performance in diverse
  quantization settings while maintaining the computational efficiency of PTQ by
  efficiently optimizing various quantization parameters. OmniQuant comprises two
  innovative components including Learnable Weight Clipping (LWC) and Learnable
  Equivalent Transformation (LET). LWC modulates the extreme values of weights by
  optimizing the clipping threshold. Meanwhile, LET tackles activation outliers
  by shifting the challenge of quantization from activations to weights through a
  learnable equivalent transformation. Operating within a differentiable
  framework using block-wise error minimization, OmniQuant can optimize the
  quantization process efficiently for both weight-only and weight-activation
  quantization. For instance, the LLaMA-2 model family with the size of 7-70B can
  be processed with OmniQuant on a single A100-40G GPU within 1-16 hours using
  128 samples. Extensive experiments validate OmniQuant's superior performance
  across diverse quantization configurations such as W4A4, W6A6, W4A16, W3A16,
  and W2A16. Additionally, OmniQuant demonstrates effectiveness in
  instruction-tuned models and delivers notable improvements in inference speed
  and memory reduction on real devices. Codes and models are available at
  https://github.com/OpenGVLab/OmniQuant.
paper_ai_keywords: Large language models (LLMs) post-training quantization (PTQ) memory footprint computational efficiency Omnidirectionally calibrated Quantization (OmniQuant) Learnable Weight Clipping (LWC) Learnable Equivalent Transformation (LET) clipping threshold activation outliers differentiable framework block-wise error minimization weight-only quantization weight-activation quantization LLaMA-2 model family A100-40G GPU W4A4 W6A6 W4A16 W3A16 W2A16 instruction-tuned models inference speed