I&S-ViT: An Inclusive & Stable Method for Pushing the Limit of
  Post-Training ViTs Quantization
Albeit the scalable performance of vision transformers (ViTs), the dense
computational costs (training & inference) undermine their position in
industrial applications. Post-training quantization (PTQ), tuning ViTs with a
tiny dataset and running in a low-bit format, well addresses the cost issue but
unluckily bears more performance drops in lower-bit cases. In this paper, we
introduce I&S-ViT, a novel method that regulates the PTQ of ViTs in an
inclusive and stable fashion. I&S-ViT first identifies two issues in the PTQ of
ViTs: (1) Quantization inefficiency in the prevalent log2 quantizer for
post-Softmax activations; (2) Rugged and magnified loss landscape in
coarse-grained quantization granularity for post-LayerNorm activations. Then,
I&S-ViT addresses these issues by introducing: (1) A novel shift-uniform-log2
quantizer (SULQ) that incorporates a shift mechanism followed by uniform
quantization to achieve both an inclusive domain representation and accurate
distribution approximation; (2) A three-stage smooth optimization strategy
(SOS) that amalgamates the strengths of channel-wise and layer-wise
quantization to enable stable learning. Comprehensive evaluations across
diverse vision tasks validate I&S-ViT' superiority over existing PTQ of ViTs
methods, particularly in low-bit scenarios. For instance, I&S-ViT elevates the
performance of 3-bit ViT-B by an impressive 50.68%.
655ac6ecdeee83130a471f9d
Yunshan Zhong
655ac6ecdeee83130a471f9e
Jiawei Hu
655ac6ecdeee83130a471f9f
Mingbao Lin
admin_assigned
2023-11-20T13:14:08.740Z
64a2a8127adb12be606ec33e
/avatars/f85dc39a23727a4d50f8a5f5a3865b0d.svg
Mingbao Lin
user
mingbao
655ac6ecdeee83130a471fa0
Mengzhao Chen
admin_assigned
2023-11-20T13:14:15.702Z
64aea082704210bf815e7551
/avatars/5c8dc0df57596c526b2bccea21835f53.svg
Mengzhao Chen
user
ChenMnZ
655ac6ecdeee83130a471fa1
Rongrong Ji
I&S-ViT: An Inclusive & Stable Method for Pushing the Limit of
  Post-Training ViTs Quantization
Albeit the scalable performance of vision transformers (ViTs), the dense
computational costs (training & inference) undermine their position in
industrial applications. Post-training quantization (PTQ), tuning ViTs with a
tiny dataset and running in a low-bit format, well addresses the cost issue but
unluckily bears more performance drops in lower-bit cases. In this paper, we
introduce I&S-ViT, a novel method that regulates the PTQ of ViTs in an
inclusive and stable fashion. I&S-ViT first identifies two issues in the PTQ of
ViTs: (1) Quantization inefficiency in the prevalent log2 quantizer for
post-Softmax activations; (2) Rugged and magnified loss landscape in
coarse-grained quantization granularity for post-LayerNorm activations. Then,
I&S-ViT addresses these issues by introducing: (1) A novel shift-uniform-log2
quantizer (SULQ) that incorporates a shift mechanism followed by uniform
quantization to achieve both an inclusive domain representation and accurate
distribution approximation; (2) A three-stage smooth optimization strategy
(SOS) that amalgamates the strengths of channel-wise and layer-wise
quantization to enable stable learning. Comprehensive evaluations across
diverse vision tasks validate I&S-ViT' superiority over existing PTQ of ViTs
methods, particularly in low-bit scenarios. For instance, I&S-ViT elevates the
performance of 3-bit ViT-B by an impressive 50.68%.
I&S-ViT introduces methods to improve post-training quantization of Vision Transformers, specifically addressing quantization inefficiency and rugged loss landscapes to maintain performance in low-bit scenarios.
vision transformers ViTs post-training quantization PTQ dense computational costs log2 quantizer post-Softmax activations LayerNorm activations coarse-grained quantization quantization inefficiency inclusive domain representation accurate distribution approximation shift-uniform-log2 quantizer SULQ three-stage smooth optimization strategy SOS channel-wise quantization layer-wise quantization stable learning vision tasks low-bit scenarios