title: FlatQuant: Flatness Matters for LLM Quantization
summary: Recently, quantization has been widely used for the compression and
  acceleration of large language models~(LLMs). Due to the outliers in LLMs, it
  is crucial to flatten weights and activations to minimize quantization error
  with the equally spaced quantization points. Prior research explores various
  pre-quantization transformations to suppress outliers, such as per-channel
  scaling and Hadamard transformation. However, we observe that these transformed
  weights and activations can still remain steep and outspread. In this paper, we
  propose FlatQuant (Fast and Learnable Affine Transformation), a new
  post-training quantization approach to enhance flatness of weights and
  activations. Our approach identifies optimal affine transformations tailored to
  each linear layer, calibrated in hours via a lightweight objective. To reduce
  runtime overhead, we apply Kronecker decomposition to the transformation
  matrices, and fuse all operations in FlatQuant into a single kernel. Extensive
  experiments show that FlatQuant sets up a new state-of-the-art quantization
  benchmark. For instance, it achieves less than 1% accuracy drop for
  W4A4 quantization on the LLaMA-3-70B model, surpassing SpinQuant by
  7.5%. For inference latency, FlatQuant reduces the slowdown induced
  by pre-quantization transformation from 0.26x of QuaRot to merely
  0.07x, bringing up to 2.3x speedup for prefill and
  1.7x speedup for decoding, respectively. Code is available at:
  https://github.com/ruikangliu/FlatQuant.
paper_authors._id: 670e0d355cde0d9af5333564
paper_authors.name: Yuxuan Sun
paper_authors._id: 670e0d355cde0d9af5333565
paper_authors.name: Ruikang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T08:07:45.453Z
paper_authors._id: 6411c22dd52c57f628f7c331
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678885386428-noauth.jpeg
paper_authors.fullname: ruikang liu
paper_authors.type: user
paper_authors.user: ruikangliu
paper_authors._id: 670e0d355cde0d9af5333566
paper_authors.name: Haoli Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:56:22.165Z
paper_authors._id: 667e1a0fa0ffad3dbc60df9d
paper_authors.avatarUrl: /avatars/e16c7a3620ae5c82b7b636296a975ae1.svg
paper_authors.fullname: Haoli Bai
paper_authors.type: user
paper_authors.user: baihaoli
paper_authors._id: 670e0d355cde0d9af5333567
paper_authors.name: Han Bao
paper_authors._id: 670e0d355cde0d9af5333568
paper_authors.name: Kang Zhao
paper_authors._id: 670e0d355cde0d9af5333569
paper_authors.name: Yuening Li
paper_authors._id: 670e0d355cde0d9af533356a
paper_authors.name: Jiaxin Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:56:35.178Z
paper_authors._id: 65268cb23427b73cb325a0b2
paper_authors.avatarUrl: /avatars/29b5cca4182aebaf11250d6250c8e278.svg
paper_authors.fullname: hu
paper_authors.type: user
paper_authors.user: jiaxinhu
paper_authors._id: 670e0d355cde0d9af533356b
paper_authors.name: Xianzhi Yu
paper_authors._id: 670e0d355cde0d9af533356c
paper_authors.name: Lu Hou
paper_authors._id: 670e0d355cde0d9af533356d
paper_authors.name: Chun Yuan
paper_authors._id: 670e0d355cde0d9af533356e
paper_authors.name: Xin Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:57:12.223Z
paper_authors._id: 647415007afa69c3c7a98f1f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647415007afa69c3c7a98f1f/pEl0PozmzNK8_PwUMiikd.jpeg
paper_authors.fullname: Xin Jiang
paper_authors.type: user
paper_authors.user: horiz94
paper_authors._id: 670e0d355cde0d9af533356f
paper_authors.name: Wulong Liu
paper_authors._id: 670e0d355cde0d9af5333570
paper_authors.name: Jun Yao
paper_title: FlatQuant: Flatness Matters for LLM Quantization
paper_summary: Recently, quantization has been widely used for the compression and
  acceleration of large language models~(LLMs). Due to the outliers in LLMs, it
  is crucial to flatten weights and activations to minimize quantization error
  with the equally spaced quantization points. Prior research explores various
  pre-quantization transformations to suppress outliers, such as per-channel
  scaling and Hadamard transformation. However, we observe that these transformed
  weights and activations can still remain steep and outspread. In this paper, we
  propose FlatQuant (Fast and Learnable Affine Transformation), a new
  post-training quantization approach to enhance flatness of weights and
  activations. Our approach identifies optimal affine transformations tailored to
  each linear layer, calibrated in hours via a lightweight objective. To reduce
  runtime overhead, we apply Kronecker decomposition to the transformation
  matrices, and fuse all operations in FlatQuant into a single kernel. Extensive
  experiments show that FlatQuant sets up a new state-of-the-art quantization
  benchmark. For instance, it achieves less than 1% accuracy drop for
  W4A4 quantization on the LLaMA-3-70B model, surpassing SpinQuant by
  7.5%. For inference latency, FlatQuant reduces the slowdown induced
  by pre-quantization transformation from 0.26x of QuaRot to merely
  0.07x, bringing up to 2.3x speedup for prefill and
  1.7x speedup for decoding, respectively. Code is available at:
  https://github.com/ruikangliu/FlatQuant.
paper_ai_summary: FlatQuant, a novel post-training quantization method, achieves better accuracy and faster inference compared to existing methods by finding optimal affine transformations and leveraging Kronecker decomposition.
paper_ai_keywords: quantization large language models (LLMs) flattened weights and activations per-channel scaling Hadamard transformation post-training quantization affine transformations Kronecker decomposition W4A4 quantization QuaRot SpinQuant