Low-Precision Training of Large Language Models: Methods, Challenges,
  and Opportunities
Large language models (LLMs) have achieved impressive performance across
various domains. However, the substantial hardware resources required for their
training present a significant barrier to efficiency and scalability. To
mitigate this challenge, low-precision training techniques have been widely
adopted, leading to notable advancements in training efficiency. Despite these
gains, low-precision training involves several componentsx2013such
as weights, activations, and gradientsx2013each of which can be
represented in different numerical formats. The resulting diversity has created
a fragmented landscape in low-precision training research, making it difficult
for researchers to gain a unified overview of the field. This survey provides a
comprehensive review of existing low-precision training methods. To
systematically organize these approaches, we categorize them into three primary
groups based on their underlying numerical formats, which is a key factor
influencing hardware compatibility, computational efficiency, and ease of
reference for readers. The categories are: (1) fixed-point and integer-based
methods, (2) floating-point-based methods, and (3) customized format-based
methods. Additionally, we discuss quantization-aware training approaches, which
share key similarities with low-precision training during forward propagation.
Finally, we highlight several promising research directions to advance this
field. A collection of papers discussed in this survey is provided in
https://github.com/Hao840/Awesome-Low-Precision-Training.
68196e23d9cad0bb5c90dd9b
Zhiwei Hao
claimed_verified
2025-05-06T08:34:14.150Z
64a62e3302e46deb19a7937e
/avatars/43553a80f2c5f6c91742c4ce2d23fe21.svg
Zhiwei Hao
user
Zhiwei840
68196e23d9cad0bb5c90dd9c
Jianyuan Guo
admin_assigned
2025-05-06T09:08:12.386Z
65c4a574d2db41f74ab2a808
/avatars/997a8a51996e909eeb318dc592b6c67a.svg
Jianyuan Guo
user
GGJY
68196e23d9cad0bb5c90dd9d
Li Shen
68196e23d9cad0bb5c90dd9e
Yong Luo
admin_assigned
2025-05-06T09:08:19.043Z
6306dc1fd37ce67e0e53c202
/avatars/d53a29925511a516495b1597fd5dc764.svg
Yong Luo
user
csdvT
68196e23d9cad0bb5c90dd9f
Han Hu
68196e23d9cad0bb5c90dda0
Guoxia Wang
admin_assigned
2025-05-06T09:08:27.139Z
662520a75480987954af60b5
/avatars/75d2509d21901c4bb187e93b23540e19.svg
Guoxia Wang
user
Guoxia
68196e23d9cad0bb5c90dda1
Dianhai Yu
68196e23d9cad0bb5c90dda2
Yonggang Wen
68196e23d9cad0bb5c90dda3
Dacheng Tao
Low-Precision Training of Large Language Models: Methods, Challenges,
  and Opportunities
Large language models (LLMs) have achieved impressive performance across
various domains. However, the substantial hardware resources required for their
training present a significant barrier to efficiency and scalability. To
mitigate this challenge, low-precision training techniques have been widely
adopted, leading to notable advancements in training efficiency. Despite these
gains, low-precision training involves several componentsx2013such
as weights, activations, and gradientsx2013each of which can be
represented in different numerical formats. The resulting diversity has created
a fragmented landscape in low-precision training research, making it difficult
for researchers to gain a unified overview of the field. This survey provides a
comprehensive review of existing low-precision training methods. To
systematically organize these approaches, we categorize them into three primary
groups based on their underlying numerical formats, which is a key factor
influencing hardware compatibility, computational efficiency, and ease of
reference for readers. The categories are: (1) fixed-point and integer-based
methods, (2) floating-point-based methods, and (3) customized format-based
methods. Additionally, we discuss quantization-aware training approaches, which
share key similarities with low-precision training during forward propagation.
Finally, we highlight several promising research directions to advance this
field. A collection of papers discussed in this survey is provided in
https://github.com/Hao840/Awesome-Low-Precision-Training.
This survey reviews low-precision training techniques for large language models, categorizing them into fixed-point, floating-point, and customized format methods, and discusses quantization-aware training.
fixed-point integer-based methods floating-point-based methods customized format-based methods quantization-aware training