FlatQuant: Flatness Matters for LLM Quantization
Recently, quantization has been widely used for the compression and
acceleration of large language models~(LLMs). Due to the outliers in LLMs, it
is crucial to flatten weights and activations to minimize quantization error
with the equally spaced quantization points. Prior research explores various
pre-quantization transformations to suppress outliers, such as per-channel
scaling and Hadamard transformation. However, we observe that these transformed
weights and activations can still remain steep and outspread. In this paper, we
propose FlatQuant (Fast and Learnable Affine Transformation), a new
post-training quantization approach to enhance flatness of weights and
activations. Our approach identifies optimal affine transformations tailored to
each linear layer, calibrated in hours via a lightweight objective. To reduce
runtime overhead, we apply Kronecker decomposition to the transformation
matrices, and fuse all operations in FlatQuant into a single kernel. Extensive
experiments show that FlatQuant sets up a new state-of-the-art quantization
benchmark. For instance, it achieves less than 1% accuracy drop for
W4A4 quantization on the LLaMA-3-70B model, surpassing SpinQuant by
7.5%. For inference latency, FlatQuant reduces the slowdown induced
by pre-quantization transformation from 0.26x of QuaRot to merely
0.07x, bringing up to 2.3x speedup for prefill and
1.7x speedup for decoding, respectively. Code is available at:
https://github.com/ruikangliu/FlatQuant.
670e0d355cde0d9af5333564
Yuxuan Sun
670e0d355cde0d9af5333565
Ruikang Liu
claimed_verified
2024-10-15T08:07:45.453Z
6411c22dd52c57f628f7c331
https://cdn-avatars.huggingface.co/v1/production/uploads/1678885386428-noauth.jpeg
ruikang liu
user
ruikangliu
670e0d355cde0d9af5333566
Haoli Bai
admin_assigned
2024-10-18T12:56:22.165Z
667e1a0fa0ffad3dbc60df9d
/avatars/e16c7a3620ae5c82b7b636296a975ae1.svg
Haoli Bai
user
baihaoli
670e0d355cde0d9af5333567
Han Bao
670e0d355cde0d9af5333568
Kang Zhao
670e0d355cde0d9af5333569
Yuening Li
670e0d355cde0d9af533356a
Jiaxin Hu
admin_assigned
2024-10-18T12:56:35.178Z
65268cb23427b73cb325a0b2
/avatars/29b5cca4182aebaf11250d6250c8e278.svg
hu
user
jiaxinhu
670e0d355cde0d9af533356b
Xianzhi Yu
670e0d355cde0d9af533356c
Lu Hou
670e0d355cde0d9af533356d
Chun Yuan
670e0d355cde0d9af533356e
Xin Jiang
admin_assigned
2024-10-18T12:57:12.223Z
647415007afa69c3c7a98f1f
https://cdn-avatars.huggingface.co/v1/production/uploads/647415007afa69c3c7a98f1f/pEl0PozmzNK8_PwUMiikd.jpeg
Xin Jiang
user
horiz94
670e0d355cde0d9af533356f
Wulong Liu
670e0d355cde0d9af5333570
Jun Yao
FlatQuant: Flatness Matters for LLM Quantization
Recently, quantization has been widely used for the compression and
acceleration of large language models~(LLMs). Due to the outliers in LLMs, it
is crucial to flatten weights and activations to minimize quantization error
with the equally spaced quantization points. Prior research explores various
pre-quantization transformations to suppress outliers, such as per-channel
scaling and Hadamard transformation. However, we observe that these transformed
weights and activations can still remain steep and outspread. In this paper, we
propose FlatQuant (Fast and Learnable Affine Transformation), a new
post-training quantization approach to enhance flatness of weights and
activations. Our approach identifies optimal affine transformations tailored to
each linear layer, calibrated in hours via a lightweight objective. To reduce
runtime overhead, we apply Kronecker decomposition to the transformation
matrices, and fuse all operations in FlatQuant into a single kernel. Extensive
experiments show that FlatQuant sets up a new state-of-the-art quantization
benchmark. For instance, it achieves less than 1% accuracy drop for
W4A4 quantization on the LLaMA-3-70B model, surpassing SpinQuant by
7.5%. For inference latency, FlatQuant reduces the slowdown induced
by pre-quantization transformation from 0.26x of QuaRot to merely
0.07x, bringing up to 2.3x speedup for prefill and
1.7x speedup for decoding, respectively. Code is available at:
https://github.com/ruikangliu/FlatQuant.
FlatQuant, a novel post-training quantization method, achieves better accuracy and faster inference compared to existing methods by finding optimal affine transformations and leveraging Kronecker decomposition.
quantization large language models (LLMs) flattened weights and activations per-channel scaling Hadamard transformation post-training quantization affine transformations Kronecker decomposition W4A4 quantization QuaRot SpinQuant