Optimizing Large Language Model Training Using FP4 Quantization
The growing computational demands of training large language models (LLMs)
necessitate more efficient methods. Quantized training presents a promising
solution by enabling low-bit arithmetic operations to reduce these costs. While
FP8 precision has demonstrated feasibility, leveraging FP4 remains a challenge
due to significant quantization errors and limited representational capacity.
This work introduces the first FP4 training framework for LLMs, addressing
these challenges with two key innovations: a differentiable quantization
estimator for precise weight updates and an outlier clamping and compensation
strategy to prevent activation collapse. To ensure stability, the framework
integrates a mixed-precision training scheme and vector-wise quantization.
Experimental results demonstrate that our FP4 framework achieves accuracy
comparable to BF16 and FP8, with minimal degradation, scaling effectively to
13B-parameter LLMs trained on up to 100B tokens. With the emergence of
next-generation hardware supporting FP4, our framework sets a foundation for
efficient ultra-low precision training.
6799b367d30dc065a2d51592
Ruizhe Wang
claimed_verified
2025-01-30T08:40:31.349Z
63203d4e260e691cfc19fcb1
/avatars/72437259c73cc4a950a2e84141097310.svg
Ruizhe Wang
user
Mr-Philo
6799b367d30dc065a2d51593
Yeyun Gong
6799b367d30dc065a2d51594
Xiao Liu
claimed_verified
2025-01-29T17:16:05.045Z
63fb6e281b4b1bd4e7ffc5be
https://cdn-avatars.huggingface.co/v1/production/uploads/1677422062937-noauth.jpeg
Xiao Liu
user
lx865712528
6799b367d30dc065a2d51595
Guoshuai Zhao
admin_assigned
2025-01-29T17:20:36.800Z
663de80ca920d195191807da
/avatars/2437ce3fa073a07b971d370c26c7ab65.svg
Guoshuai Zhao
user
crayonshine
6799b367d30dc065a2d51596
Ziyue Yang
admin_assigned
2025-01-29T17:20:42.213Z
62f6a9add3bdacb7eec0d4f5
https://cdn-avatars.huggingface.co/v1/production/uploads/1660332390183-noauth.jpeg
Ziyue Yang
user
ziyueyang37
6799b367d30dc065a2d51597
Baining Guo
6799b367d30dc065a2d51598
Zhengjun Zha
6799b367d30dc065a2d51599
Peng Cheng
extracted_pending
2025-01-29T04:49:44.372Z
653feb7ccf1f9c88f4928910
/avatars/23a6a6818116683ea9485e1470a0062f.svg
Peng Cheng
user
cp5555
Optimizing Large Language Model Training Using FP4 Quantization
The growing computational demands of training large language models (LLMs)
necessitate more efficient methods. Quantized training presents a promising
solution by enabling low-bit arithmetic operations to reduce these costs. While
FP8 precision has demonstrated feasibility, leveraging FP4 remains a challenge
due to significant quantization errors and limited representational capacity.
This work introduces the first FP4 training framework for LLMs, addressing
these challenges with two key innovations: a differentiable quantization
estimator for precise weight updates and an outlier clamping and compensation
strategy to prevent activation collapse. To ensure stability, the framework
integrates a mixed-precision training scheme and vector-wise quantization.
Experimental results demonstrate that our FP4 framework achieves accuracy
comparable to BF16 and FP8, with minimal degradation, scaling effectively to
13B-parameter LLMs trained on up to 100B tokens. With the emergence of
next-generation hardware supporting FP4, our framework sets a foundation for
efficient ultra-low precision training.
The FP4 training framework for LLMs includes differentiable quantization and outlier clamping to achieve accuracy comparable to BF16 and FP8 with minimal degradation on large-scale models.
quantized training FP4 LLMs FP8 differentiable quantization outlier clamping activation collapse mixed-precision training vector-wise quantization BF16 parameter-efficient token-wise training