title: Scaling Laws for Floating Point Quantization Training
summary: Low-precision training is considered an effective strategy for reducing both
  training and downstream inference costs. Previous scaling laws for precision
  mainly focus on integer quantization, which pay less attention to the
  constituents in floating-point quantization and thus cannot well fit the LLM
  losses in this scenario. In contrast, while floating-point quantization
  training is more commonly implemented in production, the research on it has
  been relatively superficial. In this paper, we thoroughly explore the effects
  of floating-point quantization targets, exponent bits, mantissa bits, and the
  calculation granularity of the scaling factor in floating-point quantization
  training performance of LLM models. While presenting an accurate floating-point
  quantization unified scaling law, we also provide valuable suggestions for the
  community: (1) Exponent bits contribute slightly more to the model performance
  than mantissa bits. We provide the optimal exponent-mantissa bit ratio for
  different bit numbers, which is available for future reference by hardware
  manufacturers; (2) We discover the formation of the critical data size in
  low-precision LLM training. Too much training data exceeding the critical data
  size will inversely bring in degradation of LLM performance; (3) The optimal
  floating-point quantization precision is directly proportional to the
  computational power, but within a wide computational power range, we estimate
  that the best cost-performance precision lies between 4-8 bits.
paper_authors._id: 677d1abbac2db4c22667f64a
paper_authors.name: Xingwu Sun
paper_authors._id: 677d1abbac2db4c22667f64b
paper_authors.name: Shuaipeng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T13:27:47.517Z
paper_authors._id: 64e32fb7e5bd41dd056bc943
paper_authors.avatarUrl: /avatars/fbe84d0641dfdd665cffb1aad464e8a0.svg
paper_authors.fullname: Shuaipeng Li
paper_authors.type: user
paper_authors.user: unlimblue
paper_authors._id: 677d1abbac2db4c22667f64c
paper_authors.name: Ruobing Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T13:01:47.778Z
paper_authors._id: 6622443b9b0614a760dd8123
paper_authors.avatarUrl: /avatars/acb6c1c9c429af1112530dcf76a8e420.svg
paper_authors.fullname: Ruobing Xie
paper_authors.type: user
paper_authors.user: Ruobing-Xie
paper_authors._id: 677d1abbac2db4c22667f64d
paper_authors.name: Weidong Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T13:27:58.222Z
paper_authors._id: 615c2f8416284d83200e1105
paper_authors.avatarUrl: /avatars/8e0c89925f714de58f0fc77265acd76f.svg
paper_authors.fullname: han weidong
paper_authors.type: user
paper_authors.user: dongdong2021
paper_authors._id: 677d1abbac2db4c22667f64e
paper_authors.name: Kan Wu
paper_authors._id: 677d1abbac2db4c22667f64f
paper_authors.name: Zhen Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-09T10:07:17.836Z
paper_authors._id: 62c4057732fa66fedacca0db
paper_authors.avatarUrl: /avatars/813e8c9fcd14dde1fdd415408f61bec2.svg
paper_authors.fullname: AndyYang
paper_authors.type: user
paper_authors.user: andyyang
paper_authors._id: 677d1abbac2db4c22667f650
paper_authors.name: Yixing Li
paper_authors._id: 677d1abbac2db4c22667f651
paper_authors.name: An Wang
paper_authors._id: 677d1abbac2db4c22667f652
paper_authors.name: Shuai Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-16T11:23:00.729Z
paper_authors._id: 6525511b3f8d02205bf1e9ef
paper_authors.avatarUrl: /avatars/7dfc3020337f3c403ecf133fa312b5d1.svg
paper_authors.fullname: Shuai Li
paper_authors.type: user
paper_authors.user: DiveBlue
paper_authors._id: 677d1abbac2db4c22667f653
paper_authors.name: Jinbao Xue
paper_authors._id: 677d1abbac2db4c22667f654
paper_authors.name: Yu Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-23T15:05:12.577Z
paper_authors._id: 67017abfe4d49b157ac534d9
paper_authors.avatarUrl: /avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
paper_authors.fullname: Yu Cheng
paper_authors.type: user
paper_authors.user: ych133
paper_authors._id: 677d1abbac2db4c22667f655
paper_authors.name: Yangyu Tao
paper_authors._id: 677d1abbac2db4c22667f656
paper_authors.name: Zhanhui Kang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T13:30:50.613Z
paper_authors._id: 6728117b71b9baba45c25c35
paper_authors.avatarUrl: /avatars/5e4603f00a426c6c41e3e6fef5fa0362.svg
paper_authors.fullname: zhanhui kang
paper_authors.type: user
paper_authors.user: kangzhanhui
paper_authors._id: 677d1abbac2db4c22667f657
paper_authors.name: Chengzhong Xu
paper_authors._id: 677d1abbac2db4c22667f658
paper_authors.name: Di Wang
paper_authors._id: 677d1abbac2db4c22667f659
paper_authors.name: Jie Jiang
paper_title: Scaling Laws for Floating Point Quantization Training
paper_summary: Low-precision training is considered an effective strategy for reducing both
  training and downstream inference costs. Previous scaling laws for precision
  mainly focus on integer quantization, which pay less attention to the
  constituents in floating-point quantization and thus cannot well fit the LLM
  losses in this scenario. In contrast, while floating-point quantization
  training is more commonly implemented in production, the research on it has
  been relatively superficial. In this paper, we thoroughly explore the effects
  of floating-point quantization targets, exponent bits, mantissa bits, and the
  calculation granularity of the scaling factor in floating-point quantization
  training performance of LLM models. While presenting an accurate floating-point
  quantization unified scaling law, we also provide valuable suggestions for the
  community: (1) Exponent bits contribute slightly more to the model performance
  than mantissa bits. We provide the optimal exponent-mantissa bit ratio for
  different bit numbers, which is available for future reference by hardware
  manufacturers; (2) We discover the formation of the critical data size in
  low-precision LLM training. Too much training data exceeding the critical data
  size will inversely bring in degradation of LLM performance; (3) The optimal
  floating-point quantization precision is directly proportional to the
  computational power, but within a wide computational power range, we estimate
  that the best cost-performance precision lies between 4-8 bits.
paper_ai_keywords: low-precision training integer quantization floating-point quantization LLM losses scaling factor floating-point quantization unified scaling law exponent bits mantissa bits calculation granularity model performance critical data size computational power cost-performance precision