FP8-LM: Training FP8 Large Language Models
In this paper, we explore FP8 low-bit data formats for efficient training of
large language models (LLMs). Our key insight is that most variables, such as
gradients and optimizer states, in LLM training can employ low-precision data
formats without compromising model accuracy and requiring no changes to
hyper-parameters. Specifically, we propose a new FP8 automatic mixed-precision
framework for training LLMs. This framework offers three levels of FP8
utilization to streamline mixed-precision and distributed parallel training for
LLMs. It gradually incorporates 8-bit gradients, optimizer states, and
distributed learning in an incremental manner. Experiment results show that,
during the training of GPT-175B model on H100 GPU platform, our FP8
mixed-precision training framework not only achieved a remarkable 42% reduction
in real memory usage but also ran 64% faster than the widely adopted BF16
framework (i.e., Megatron-LM), surpassing the speed of Nvidia Transformer
Engine by 17%. This largely reduces the training costs for large foundation
models. Furthermore, our FP8 mixed-precision training methodology is generic.
It can be seamlessly applied to other tasks such as LLM instruction tuning and
reinforcement learning with human feedback, offering savings in fine-tuning
expenses. Our FP8 low-precision training framework is open-sourced at
{https://github.com/Azure/MS-AMP}{aka.ms/MS.AMP}.
653f175be8ed050cb3513a27
Houwen Peng
admin_assigned
2023-10-30T13:20:04.911Z
631952b6f18d0b5d999ca397
/avatars/eb87dfc142a7602f8fb888f7b7b60d38.svg
Peng
user
Houwen
653f175be8ed050cb3513a28
Kan Wu
claimed_verified
2023-10-30T13:20:14.684Z
64acfc3d7a9df1bd414824d5
https://cdn-avatars.huggingface.co/v1/production/uploads/64acfc3d7a9df1bd414824d5/cMnQJ5ObNeHhmXvW7GMfc.png
Jackie Wu
user
wkcn
653f175be8ed050cb3513a29
Yixuan Wei
admin_assigned
2023-10-30T13:20:24.646Z
640efe54a92fedb0e84ee598
/avatars/cb1c6720525bfdbf59a9d3eac93a879e.svg
Yixuan Wei
user
EasonWei
653f175be8ed050cb3513a2a
Guoshuai Zhao
653f175be8ed050cb3513a2b
Yuxiang Yang
653f175be8ed050cb3513a2c
Ze Liu
admin_assigned
2023-10-30T13:21:44.217Z
61ff8a090fec6e1502ca2d89
/avatars/1f72372d4e4264592a4a573c30c3ab3a.svg
Ze Liu
user
zeliu98
653f175be8ed050cb3513a2d
Yifan Xiong
admin_assigned
2023-10-30T13:21:53.757Z
6501267aaf1e4021439cbc1d
/avatars/cd107d1a887e292c2d107748eadcac25.svg
YIfan Xiong
user
AGcwh
653f175be8ed050cb3513a2e
Ziyue Yang
653f175be8ed050cb3513a2f
Bolin Ni
admin_assigned
2023-10-30T13:22:40.163Z
642149592cc2b3c39e7ef6c9
/avatars/fed5d447375cd8c3b2bc9e096fe681de.svg
Bolin Ni
user
nbl97
653f175be8ed050cb3513a30
Jingcheng Hu
admin_assigned
2023-10-30T13:22:50.943Z
625026b7d2d191ac43320c5e
https://cdn-avatars.huggingface.co/v1/production/uploads/625026b7d2d191ac43320c5e/2ExzHlZ-Bk8SQMyBjeY6N.jpeg
Jingcheng Hu
user
reign12
653f175be8ed050cb3513a31
Ruihang Li
admin_assigned
2023-10-30T13:23:11.480Z
647d6d52adaf5cc26dab2a23
/avatars/55c535d6498f81db7fee29ecea626293.svg
Ruihang Liu
user
LeoLRH
653f175be8ed050cb3513a32
Miaosen Zhang
admin_assigned
2023-10-30T13:23:20.099Z
62a30bf72dac39c2173c0a8c
/avatars/15fb1ea3dcc7ccd8bc8002ce282e27b3.svg
Miaosen Zhang
user
Miaosen
653f175be8ed050cb3513a33
Chen Li
claimed_verified
2023-11-02T16:15:02.895Z
64e0debc02fa032de402feb0
/avatars/6c28a13216f7ae4c047cf6ea9846f6e3.svg
Chen Li
user
LC-Edward
653f175be8ed050cb3513a34
Jia Ning
653f175be8ed050cb3513a35
Ruizhe Wang
claimed_verified
2024-06-11T07:32:15.278Z
63203d4e260e691cfc19fcb1
/avatars/72437259c73cc4a950a2e84141097310.svg
Ruizhe Wang
user
Mr-Philo
653f175be8ed050cb3513a36
Zheng Zhang
653f175be8ed050cb3513a37
Shuguang Liu
653f175be8ed050cb3513a38
Joe Chau
653f175be8ed050cb3513a39
Han Hu
653f175be8ed050cb3513a3a
Peng Cheng
claimed_verified
2023-10-31T08:23:33.140Z
653feb7ccf1f9c88f4928910
/avatars/23a6a6818116683ea9485e1470a0062f.svg
Peng Cheng
user
cp5555
FP8-LM: Training FP8 Large Language Models
In this paper, we explore FP8 low-bit data formats for efficient training of
large language models (LLMs). Our key insight is that most variables, such as
gradients and optimizer states, in LLM training can employ low-precision data
formats without compromising model accuracy and requiring no changes to
hyper-parameters. Specifically, we propose a new FP8 automatic mixed-precision
framework for training LLMs. This framework offers three levels of FP8
utilization to streamline mixed-precision and distributed parallel training for
LLMs. It gradually incorporates 8-bit gradients, optimizer states, and
distributed learning in an incremental manner. Experiment results show that,
during the training of GPT-175B model on H100 GPU platform, our FP8
mixed-precision training framework not only achieved a remarkable 42% reduction
in real memory usage but also ran 64% faster than the widely adopted BF16
framework (i.e., Megatron-LM), surpassing the speed of Nvidia Transformer
Engine by 17%. This largely reduces the training costs for large foundation
models. Furthermore, our FP8 mixed-precision training methodology is generic.
It can be seamlessly applied to other tasks such as LLM instruction tuning and
reinforcement learning with human feedback, offering savings in fine-tuning
expenses. Our FP8 low-precision training framework is open-sourced at
{https://github.com/Azure/MS-AMP}{aka.ms/MS.AMP}.
A new FP8 automatic mixed-precision framework for training large language models reduces memory usage and increases speed compared to BF16 and Nvidia Transformer Engine.
FP8 low-bit data formats large language models LLM gradients optimizer states distributed parallel training FP8 automatic mixed-precision GPT-175B H100 GPU BF16 Nvidia Transformer Engine fine-tuning instruction tuning reinforcement learning with human feedback