Scaling Laws for Floating Point Quantization Training
Low-precision training is considered an effective strategy for reducing both
training and downstream inference costs. Previous scaling laws for precision
mainly focus on integer quantization, which pay less attention to the
constituents in floating-point quantization and thus cannot well fit the LLM
losses in this scenario. In contrast, while floating-point quantization
training is more commonly implemented in production, the research on it has
been relatively superficial. In this paper, we thoroughly explore the effects
of floating-point quantization targets, exponent bits, mantissa bits, and the
calculation granularity of the scaling factor in floating-point quantization
training performance of LLM models. While presenting an accurate floating-point
quantization unified scaling law, we also provide valuable suggestions for the
community: (1) Exponent bits contribute slightly more to the model performance
than mantissa bits. We provide the optimal exponent-mantissa bit ratio for
different bit numbers, which is available for future reference by hardware
manufacturers; (2) We discover the formation of the critical data size in
low-precision LLM training. Too much training data exceeding the critical data
size will inversely bring in degradation of LLM performance; (3) The optimal
floating-point quantization precision is directly proportional to the
computational power, but within a wide computational power range, we estimate
that the best cost-performance precision lies between 4-8 bits.
677d1abbac2db4c22667f64a
Xingwu Sun
677d1abbac2db4c22667f64b
Shuaipeng Li
admin_assigned
2025-01-07T13:27:47.517Z
64e32fb7e5bd41dd056bc943
/avatars/fbe84d0641dfdd665cffb1aad464e8a0.svg
Shuaipeng Li
user
unlimblue
677d1abbac2db4c22667f64c
Ruobing Xie
claimed_verified
2025-01-07T13:01:47.778Z
6622443b9b0614a760dd8123
/avatars/acb6c1c9c429af1112530dcf76a8e420.svg
Ruobing Xie
user
Ruobing-Xie
677d1abbac2db4c22667f64d
Weidong Han
admin_assigned
2025-01-07T13:27:58.222Z
615c2f8416284d83200e1105
/avatars/8e0c89925f714de58f0fc77265acd76f.svg
han weidong
user
dongdong2021
677d1abbac2db4c22667f64e
Kan Wu
677d1abbac2db4c22667f64f
Zhen Yang
claimed_verified
2025-01-09T10:07:17.836Z
62c4057732fa66fedacca0db
/avatars/813e8c9fcd14dde1fdd415408f61bec2.svg
Zhen Yang
user
andyyang
677d1abbac2db4c22667f650
Yixing Li
677d1abbac2db4c22667f651
An Wang
677d1abbac2db4c22667f652
Shuai Li
claimed_verified
2025-01-16T11:23:00.729Z
6525511b3f8d02205bf1e9ef
/avatars/7dfc3020337f3c403ecf133fa312b5d1.svg
Shuai Li
user
DiveBlue
677d1abbac2db4c22667f653
Jinbao Xue
677d1abbac2db4c22667f654
Yu Cheng
claimed_verified
2025-01-23T15:05:12.577Z
67017abfe4d49b157ac534d9
/avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
Yu Cheng
user
ych133
677d1abbac2db4c22667f655
Yangyu Tao
677d1abbac2db4c22667f656
Zhanhui Kang
admin_assigned
2025-01-07T13:30:50.613Z
6728117b71b9baba45c25c35
/avatars/5e4603f00a426c6c41e3e6fef5fa0362.svg
zhanhui kang
user
kangzhanhui
677d1abbac2db4c22667f657
Chengzhong Xu
677d1abbac2db4c22667f658
Di Wang
677d1abbac2db4c22667f659
Jie Jiang
Scaling Laws for Floating Point Quantization Training
Low-precision training is considered an effective strategy for reducing both
training and downstream inference costs. Previous scaling laws for precision
mainly focus on integer quantization, which pay less attention to the
constituents in floating-point quantization and thus cannot well fit the LLM
losses in this scenario. In contrast, while floating-point quantization
training is more commonly implemented in production, the research on it has
been relatively superficial. In this paper, we thoroughly explore the effects
of floating-point quantization targets, exponent bits, mantissa bits, and the
calculation granularity of the scaling factor in floating-point quantization
training performance of LLM models. While presenting an accurate floating-point
quantization unified scaling law, we also provide valuable suggestions for the
community: (1) Exponent bits contribute slightly more to the model performance
than mantissa bits. We provide the optimal exponent-mantissa bit ratio for
different bit numbers, which is available for future reference by hardware
manufacturers; (2) We discover the formation of the critical data size in
low-precision LLM training. Too much training data exceeding the critical data
size will inversely bring in degradation of LLM performance; (3) The optimal
floating-point quantization precision is directly proportional to the
computational power, but within a wide computational power range, we estimate
that the best cost-performance precision lies between 4-8 bits.
Floating-point quantization training of LLMs is explored to optimize performance and reduce costs, revealing critical insights into bit allocation, data size impacts, and precision-performance relationships.
floating-point quantization low-precision training scaling law exponent bits mantissa bits calculation granularity scaling factor critical data size computational power cost-performance precision