title: Low-Precision Training of Large Language Models: Methods, Challenges,
    and Opportunities
summary: Large language models (LLMs) have achieved impressive performance across
  various domains. However, the substantial hardware resources required for their
  training present a significant barrier to efficiency and scalability. To
  mitigate this challenge, low-precision training techniques have been widely
  adopted, leading to notable advancements in training efficiency. Despite these
  gains, low-precision training involves several componentsx2013such
  as weights, activations, and gradientsx2013each of which can be
  represented in different numerical formats. The resulting diversity has created
  a fragmented landscape in low-precision training research, making it difficult
  for researchers to gain a unified overview of the field. This survey provides a
  comprehensive review of existing low-precision training methods. To
  systematically organize these approaches, we categorize them into three primary
  groups based on their underlying numerical formats, which is a key factor
  influencing hardware compatibility, computational efficiency, and ease of
  reference for readers. The categories are: (1) fixed-point and integer-based
  methods, (2) floating-point-based methods, and (3) customized format-based
  methods. Additionally, we discuss quantization-aware training approaches, which
  share key similarities with low-precision training during forward propagation.
  Finally, we highlight several promising research directions to advance this
  field. A collection of papers discussed in this survey is provided in
  https://github.com/Hao840/Awesome-Low-Precision-Training.
paper_authors._id: 68196e23d9cad0bb5c90dd9b
paper_authors.name: Zhiwei Hao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-06T08:34:14.150Z
paper_authors._id: 64a62e3302e46deb19a7937e
paper_authors.avatarUrl: /avatars/43553a80f2c5f6c91742c4ce2d23fe21.svg
paper_authors.fullname: Zhiwei Hao
paper_authors.type: user
paper_authors.user: Zhiwei840
paper_authors._id: 68196e23d9cad0bb5c90dd9c
paper_authors.name: Jianyuan Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:08:12.386Z
paper_authors._id: 65c4a574d2db41f74ab2a808
paper_authors.avatarUrl: /avatars/997a8a51996e909eeb318dc592b6c67a.svg
paper_authors.fullname: Jianyuan Guo
paper_authors.type: user
paper_authors.user: GGJY
paper_authors._id: 68196e23d9cad0bb5c90dd9d
paper_authors.name: Li Shen
paper_authors._id: 68196e23d9cad0bb5c90dd9e
paper_authors.name: Yong Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:08:19.043Z
paper_authors._id: 6306dc1fd37ce67e0e53c202
paper_authors.avatarUrl: /avatars/d53a29925511a516495b1597fd5dc764.svg
paper_authors.fullname: Yong Luo
paper_authors.type: user
paper_authors.user: csdvT
paper_authors._id: 68196e23d9cad0bb5c90dd9f
paper_authors.name: Han Hu
paper_authors._id: 68196e23d9cad0bb5c90dda0
paper_authors.name: Guoxia Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:08:27.139Z
paper_authors._id: 662520a75480987954af60b5
paper_authors.avatarUrl: /avatars/75d2509d21901c4bb187e93b23540e19.svg
paper_authors.fullname: Guoxia Wang
paper_authors.type: user
paper_authors.user: Guoxia
paper_authors._id: 68196e23d9cad0bb5c90dda1
paper_authors.name: Dianhai Yu
paper_authors._id: 68196e23d9cad0bb5c90dda2
paper_authors.name: Yonggang Wen
paper_authors._id: 68196e23d9cad0bb5c90dda3
paper_authors.name: Dacheng Tao
paper_title: Low-Precision Training of Large Language Models: Methods, Challenges,
    and Opportunities
paper_summary: Large language models (LLMs) have achieved impressive performance across
  various domains. However, the substantial hardware resources required for their
  training present a significant barrier to efficiency and scalability. To
  mitigate this challenge, low-precision training techniques have been widely
  adopted, leading to notable advancements in training efficiency. Despite these
  gains, low-precision training involves several componentsx2013such
  as weights, activations, and gradientsx2013each of which can be
  represented in different numerical formats. The resulting diversity has created
  a fragmented landscape in low-precision training research, making it difficult
  for researchers to gain a unified overview of the field. This survey provides a
  comprehensive review of existing low-precision training methods. To
  systematically organize these approaches, we categorize them into three primary
  groups based on their underlying numerical formats, which is a key factor
  influencing hardware compatibility, computational efficiency, and ease of
  reference for readers. The categories are: (1) fixed-point and integer-based
  methods, (2) floating-point-based methods, and (3) customized format-based
  methods. Additionally, we discuss quantization-aware training approaches, which
  share key similarities with low-precision training during forward propagation.
  Finally, we highlight several promising research directions to advance this
  field. A collection of papers discussed in this survey is provided in
  https://github.com/Hao840/Awesome-Low-Precision-Training.
paper_ai_summary: This survey reviews low-precision training techniques for large language models, categorizing them into fixed-point, floating-point, and customized format methods, and discusses quantization-aware training.
paper_ai_keywords: fixed-point integer-based methods floating-point-based methods customized format-based methods quantization-aware training