title: Towards Next-Level Post-Training Quantization of Hyper-Scale
    Transformers
summary: With the increasing complexity of generative AI models, post-training
  quantization (PTQ) has emerged as a promising solution for deploying
  hyper-scale models on edge devices such as mobile devices and TVs. Existing PTQ
  schemes, however, consume considerable time and resources, which could be a
  bottleneck in real situations where frequent model updates and multiple
  hyper-parameter tunings are required. As a cost-effective alternative, one-shot
  PTQ schemes have been proposed. Still, the performance is somewhat limited
  because they cannot consider the inter-layer dependency within the attention
  module, which is a very important feature of Transformers. In this paper, we
  thus propose a novel PTQ algorithm that balances accuracy and efficiency. The
  key idea of the proposed algorithm called aespa is to perform quantization
  layer-wise for efficiency while considering cross-layer dependency to preserve
  the attention score. Through extensive experiments on various language models
  and complexity analysis, we demonstrate that aespa is accurate and efficient in
  quantizing Transformer models.
paper_authors._id: 65cda204eedafd0c26a15d2a
paper_authors.name: Junhan Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T08:03:51.680Z
paper_authors._id: 6609dcee4fa3a72a979e0769
paper_authors.avatarUrl: /avatars/588fd3a1e9d6d0952cd6cc7f7e2e8a2e.svg
paper_authors.fullname: Junhan
paper_authors.type: user
paper_authors.user: SShock92
paper_authors._id: 65cda204eedafd0c26a15d2b
paper_authors.name: Kyungphil Park
paper_authors._id: 65cda204eedafd0c26a15d2c
paper_authors.name: Chungman Lee
paper_authors._id: 65cda204eedafd0c26a15d2d
paper_authors.name: Ho-young Kim
paper_authors._id: 65cda204eedafd0c26a15d2e
paper_authors.name: Joonyoung Kim
paper_authors._id: 65cda204eedafd0c26a15d2f
paper_authors.name: Yongkweon Jeon
paper_title: Towards Next-Level Post-Training Quantization of Hyper-Scale
    Transformers
paper_summary: With the increasing complexity of generative AI models, post-training
  quantization (PTQ) has emerged as a promising solution for deploying
  hyper-scale models on edge devices such as mobile devices and TVs. Existing PTQ
  schemes, however, consume considerable time and resources, which could be a
  bottleneck in real situations where frequent model updates and multiple
  hyper-parameter tunings are required. As a cost-effective alternative, one-shot
  PTQ schemes have been proposed. Still, the performance is somewhat limited
  because they cannot consider the inter-layer dependency within the attention
  module, which is a very important feature of Transformers. In this paper, we
  thus propose a novel PTQ algorithm that balances accuracy and efficiency. The
  key idea of the proposed algorithm called aespa is to perform quantization
  layer-wise for efficiency while considering cross-layer dependency to preserve
  the attention score. Through extensive experiments on various language models
  and complexity analysis, we demonstrate that aespa is accurate and efficient in
  quantizing Transformer models.
paper_ai_keywords: post-training quantization (PTQ) hyper-scale models edge devices mobile devices TVs one-shot PTQ inter-layer dependency Transformers attention module attention score aespa algorithm quantization layer-wise language models