title: Optimizing Large Language Model Training Using FP4 Quantization
summary: The growing computational demands of training large language models (LLMs)
  necessitate more efficient methods. Quantized training presents a promising
  solution by enabling low-bit arithmetic operations to reduce these costs. While
  FP8 precision has demonstrated feasibility, leveraging FP4 remains a challenge
  due to significant quantization errors and limited representational capacity.
  This work introduces the first FP4 training framework for LLMs, addressing
  these challenges with two key innovations: a differentiable quantization
  estimator for precise weight updates and an outlier clamping and compensation
  strategy to prevent activation collapse. To ensure stability, the framework
  integrates a mixed-precision training scheme and vector-wise quantization.
  Experimental results demonstrate that our FP4 framework achieves accuracy
  comparable to BF16 and FP8, with minimal degradation, scaling effectively to
  13B-parameter LLMs trained on up to 100B tokens. With the emergence of
  next-generation hardware supporting FP4, our framework sets a foundation for
  efficient ultra-low precision training.
paper_authors._id: 6799b367d30dc065a2d51592
paper_authors.name: Ruizhe Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-30T08:40:31.349Z
paper_authors._id: 63203d4e260e691cfc19fcb1
paper_authors.avatarUrl: /avatars/72437259c73cc4a950a2e84141097310.svg
paper_authors.fullname: Ruizhe Wang
paper_authors.type: user
paper_authors.user: Mr-Philo
paper_authors._id: 6799b367d30dc065a2d51593
paper_authors.name: Yeyun Gong
paper_authors._id: 6799b367d30dc065a2d51594
paper_authors.name: Xiao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-29T17:16:05.045Z
paper_authors._id: 63fb6e281b4b1bd4e7ffc5be
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677422062937-noauth.jpeg
paper_authors.fullname: Xiao Liu
paper_authors.type: user
paper_authors.user: lx865712528
paper_authors._id: 6799b367d30dc065a2d51595
paper_authors.name: Guoshuai Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-29T17:20:36.800Z
paper_authors._id: 663de80ca920d195191807da
paper_authors.avatarUrl: /avatars/2437ce3fa073a07b971d370c26c7ab65.svg
paper_authors.fullname: Guoshuai Zhao
paper_authors.type: user
paper_authors.user: crayonshine
paper_authors._id: 6799b367d30dc065a2d51596
paper_authors.name: Ziyue Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-29T17:20:42.213Z
paper_authors._id: 62f6a9add3bdacb7eec0d4f5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660332390183-noauth.jpeg
paper_authors.fullname: Ziyue Yang
paper_authors.type: user
paper_authors.user: ziyueyang37
paper_authors._id: 6799b367d30dc065a2d51597
paper_authors.name: Baining Guo
paper_authors._id: 6799b367d30dc065a2d51598
paper_authors.name: Zhengjun Zha
paper_authors._id: 6799b367d30dc065a2d51599
paper_authors.name: Peng Cheng
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-01-29T04:49:44.372Z
paper_authors._id: 653feb7ccf1f9c88f4928910
paper_authors.avatarUrl: /avatars/23a6a6818116683ea9485e1470a0062f.svg
paper_authors.fullname: Peng Cheng
paper_authors.type: user
paper_authors.user: cp5555
paper_title: Optimizing Large Language Model Training Using FP4 Quantization
paper_summary: The growing computational demands of training large language models (LLMs)
  necessitate more efficient methods. Quantized training presents a promising
  solution by enabling low-bit arithmetic operations to reduce these costs. While
  FP8 precision has demonstrated feasibility, leveraging FP4 remains a challenge
  due to significant quantization errors and limited representational capacity.
  This work introduces the first FP4 training framework for LLMs, addressing
  these challenges with two key innovations: a differentiable quantization
  estimator for precise weight updates and an outlier clamping and compensation
  strategy to prevent activation collapse. To ensure stability, the framework
  integrates a mixed-precision training scheme and vector-wise quantization.
  Experimental results demonstrate that our FP4 framework achieves accuracy
  comparable to BF16 and FP8, with minimal degradation, scaling effectively to
  13B-parameter LLMs trained on up to 100B tokens. With the emergence of
  next-generation hardware supporting FP4, our framework sets a foundation for
  efficient ultra-low precision training.
paper_ai_keywords: quantized training low-bit arithmetic operations FP8 precision FP4 precision quantization errors representational capacity differentiable quantization estimator weight updates outlier clamping compensation strategy activation collapse mixed-precision training vector-wise quantization BF16 precision 13B-parameter LLMs 100B tokens ultra-low precision training