title: I&S-ViT: An Inclusive & Stable Method for Pushing the Limit of
    Post-Training ViTs Quantization
summary: Albeit the scalable performance of vision transformers (ViTs), the dense
  computational costs (training & inference) undermine their position in
  industrial applications. Post-training quantization (PTQ), tuning ViTs with a
  tiny dataset and running in a low-bit format, well addresses the cost issue but
  unluckily bears more performance drops in lower-bit cases. In this paper, we
  introduce I&S-ViT, a novel method that regulates the PTQ of ViTs in an
  inclusive and stable fashion. I&S-ViT first identifies two issues in the PTQ of
  ViTs: (1) Quantization inefficiency in the prevalent log2 quantizer for
  post-Softmax activations; (2) Rugged and magnified loss landscape in
  coarse-grained quantization granularity for post-LayerNorm activations. Then,
  I&S-ViT addresses these issues by introducing: (1) A novel shift-uniform-log2
  quantizer (SULQ) that incorporates a shift mechanism followed by uniform
  quantization to achieve both an inclusive domain representation and accurate
  distribution approximation; (2) A three-stage smooth optimization strategy
  (SOS) that amalgamates the strengths of channel-wise and layer-wise
  quantization to enable stable learning. Comprehensive evaluations across
  diverse vision tasks validate I&S-ViT' superiority over existing PTQ of ViTs
  methods, particularly in low-bit scenarios. For instance, I&S-ViT elevates the
  performance of 3-bit ViT-B by an impressive 50.68%.
paper_authors._id: 655ac6ecdeee83130a471f9d
paper_authors.name: Yunshan Zhong
paper_authors._id: 655ac6ecdeee83130a471f9e
paper_authors.name: Jiawei Hu
paper_authors._id: 655ac6ecdeee83130a471f9f
paper_authors.name: Mingbao Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-20T13:14:08.740Z
paper_authors._id: 64a2a8127adb12be606ec33e
paper_authors.avatarUrl: /avatars/f85dc39a23727a4d50f8a5f5a3865b0d.svg
paper_authors.fullname: Mingbao Lin
paper_authors.type: user
paper_authors.user: mingbao
paper_authors._id: 655ac6ecdeee83130a471fa0
paper_authors.name: Mengzhao Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-20T13:14:15.702Z
paper_authors._id: 64aea082704210bf815e7551
paper_authors.avatarUrl: /avatars/5c8dc0df57596c526b2bccea21835f53.svg
paper_authors.fullname: Mengzhao Chen
paper_authors.type: user
paper_authors.user: ChenMnZ
paper_authors._id: 655ac6ecdeee83130a471fa1
paper_authors.name: Rongrong Ji
paper_title: I&S-ViT: An Inclusive & Stable Method for Pushing the Limit of
    Post-Training ViTs Quantization
paper_summary: Albeit the scalable performance of vision transformers (ViTs), the dense
  computational costs (training & inference) undermine their position in
  industrial applications. Post-training quantization (PTQ), tuning ViTs with a
  tiny dataset and running in a low-bit format, well addresses the cost issue but
  unluckily bears more performance drops in lower-bit cases. In this paper, we
  introduce I&S-ViT, a novel method that regulates the PTQ of ViTs in an
  inclusive and stable fashion. I&S-ViT first identifies two issues in the PTQ of
  ViTs: (1) Quantization inefficiency in the prevalent log2 quantizer for
  post-Softmax activations; (2) Rugged and magnified loss landscape in
  coarse-grained quantization granularity for post-LayerNorm activations. Then,
  I&S-ViT addresses these issues by introducing: (1) A novel shift-uniform-log2
  quantizer (SULQ) that incorporates a shift mechanism followed by uniform
  quantization to achieve both an inclusive domain representation and accurate
  distribution approximation; (2) A three-stage smooth optimization strategy
  (SOS) that amalgamates the strengths of channel-wise and layer-wise
  quantization to enable stable learning. Comprehensive evaluations across
  diverse vision tasks validate I&S-ViT' superiority over existing PTQ of ViTs
  methods, particularly in low-bit scenarios. For instance, I&S-ViT elevates the
  performance of 3-bit ViT-B by an impressive 50.68%.
paper_ai_summary: I&S-ViT introduces methods to improve post-training quantization of Vision Transformers, specifically addressing quantization inefficiency and rugged loss landscapes to maintain performance in low-bit scenarios.
paper_ai_keywords: vision transformers ViTs post-training quantization PTQ dense computational costs log2 quantizer post-Softmax activations LayerNorm activations coarse-grained quantization quantization inefficiency inclusive domain representation accurate distribution approximation shift-uniform-log2 quantizer SULQ three-stage smooth optimization strategy SOS channel-wise quantization layer-wise quantization stable learning vision tasks low-bit scenarios