title: FP8-LM: Training FP8 Large Language Models
summary: In this paper, we explore FP8 low-bit data formats for efficient training of
  large language models (LLMs). Our key insight is that most variables, such as
  gradients and optimizer states, in LLM training can employ low-precision data
  formats without compromising model accuracy and requiring no changes to
  hyper-parameters. Specifically, we propose a new FP8 automatic mixed-precision
  framework for training LLMs. This framework offers three levels of FP8
  utilization to streamline mixed-precision and distributed parallel training for
  LLMs. It gradually incorporates 8-bit gradients, optimizer states, and
  distributed learning in an incremental manner. Experiment results show that,
  during the training of GPT-175B model on H100 GPU platform, our FP8
  mixed-precision training framework not only achieved a remarkable 42% reduction
  in real memory usage but also ran 64% faster than the widely adopted BF16
  framework (i.e., Megatron-LM), surpassing the speed of Nvidia Transformer
  Engine by 17%. This largely reduces the training costs for large foundation
  models. Furthermore, our FP8 mixed-precision training methodology is generic.
  It can be seamlessly applied to other tasks such as LLM instruction tuning and
  reinforcement learning with human feedback, offering savings in fine-tuning
  expenses. Our FP8 low-precision training framework is open-sourced at
  {https://github.com/Azure/MS-AMP}{aka.ms/MS.AMP}.
paper_authors._id: 653f175be8ed050cb3513a27
paper_authors.name: Houwen Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:20:04.911Z
paper_authors._id: 631952b6f18d0b5d999ca397
paper_authors.avatarUrl: /avatars/eb87dfc142a7602f8fb888f7b7b60d38.svg
paper_authors.fullname: Peng
paper_authors.type: user
paper_authors.user: Houwen
paper_authors._id: 653f175be8ed050cb3513a28
paper_authors.name: Kan Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-30T13:20:14.684Z
paper_authors._id: 64acfc3d7a9df1bd414824d5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64acfc3d7a9df1bd414824d5/cMnQJ5ObNeHhmXvW7GMfc.png
paper_authors.fullname: Jackie Wu
paper_authors.type: user
paper_authors.user: wkcn
paper_authors._id: 653f175be8ed050cb3513a29
paper_authors.name: Yixuan Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:20:24.646Z
paper_authors._id: 640efe54a92fedb0e84ee598
paper_authors.avatarUrl: /avatars/cb1c6720525bfdbf59a9d3eac93a879e.svg
paper_authors.fullname: Yixuan Wei
paper_authors.type: user
paper_authors.user: EasonWei
paper_authors._id: 653f175be8ed050cb3513a2a
paper_authors.name: Guoshuai Zhao
paper_authors._id: 653f175be8ed050cb3513a2b
paper_authors.name: Yuxiang Yang
paper_authors._id: 653f175be8ed050cb3513a2c
paper_authors.name: Ze Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:21:44.217Z
paper_authors._id: 61ff8a090fec6e1502ca2d89
paper_authors.avatarUrl: /avatars/1f72372d4e4264592a4a573c30c3ab3a.svg
paper_authors.fullname: Ze Liu
paper_authors.type: user
paper_authors.user: zeliu98
paper_authors._id: 653f175be8ed050cb3513a2d
paper_authors.name: Yifan Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:21:53.757Z
paper_authors._id: 6501267aaf1e4021439cbc1d
paper_authors.avatarUrl: /avatars/cd107d1a887e292c2d107748eadcac25.svg
paper_authors.fullname: YIfan Xiong
paper_authors.type: user
paper_authors.user: AGcwh
paper_authors._id: 653f175be8ed050cb3513a2e
paper_authors.name: Ziyue Yang
paper_authors._id: 653f175be8ed050cb3513a2f
paper_authors.name: Bolin Ni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:22:40.163Z
paper_authors._id: 642149592cc2b3c39e7ef6c9
paper_authors.avatarUrl: /avatars/fed5d447375cd8c3b2bc9e096fe681de.svg
paper_authors.fullname: Bolin Ni
paper_authors.type: user
paper_authors.user: nbl97
paper_authors._id: 653f175be8ed050cb3513a30
paper_authors.name: Jingcheng Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:22:50.943Z
paper_authors._id: 625026b7d2d191ac43320c5e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/625026b7d2d191ac43320c5e/2ExzHlZ-Bk8SQMyBjeY6N.jpeg
paper_authors.fullname: Jingcheng Hu
paper_authors.type: user
paper_authors.user: reign12
paper_authors._id: 653f175be8ed050cb3513a31
paper_authors.name: Ruihang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:23:11.480Z
paper_authors._id: 647d6d52adaf5cc26dab2a23
paper_authors.avatarUrl: /avatars/55c535d6498f81db7fee29ecea626293.svg
paper_authors.fullname: Ruihang Liu
paper_authors.type: user
paper_authors.user: LeoLRH
paper_authors._id: 653f175be8ed050cb3513a32
paper_authors.name: Miaosen Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:23:20.099Z
paper_authors._id: 62a30bf72dac39c2173c0a8c
paper_authors.avatarUrl: /avatars/15fb1ea3dcc7ccd8bc8002ce282e27b3.svg
paper_authors.fullname: Miaosen Zhang
paper_authors.type: user
paper_authors.user: Miaosen
paper_authors._id: 653f175be8ed050cb3513a33
paper_authors.name: Chen Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-02T16:15:02.895Z
paper_authors._id: 64e0debc02fa032de402feb0
paper_authors.avatarUrl: /avatars/6c28a13216f7ae4c047cf6ea9846f6e3.svg
paper_authors.fullname: Chen Li
paper_authors.type: user
paper_authors.user: LC-Edward
paper_authors._id: 653f175be8ed050cb3513a34
paper_authors.name: Jia Ning
paper_authors._id: 653f175be8ed050cb3513a35
paper_authors.name: Ruizhe Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-11T07:32:15.278Z
paper_authors._id: 63203d4e260e691cfc19fcb1
paper_authors.avatarUrl: /avatars/72437259c73cc4a950a2e84141097310.svg
paper_authors.fullname: Ruizhe Wang
paper_authors.type: user
paper_authors.user: Mr-Philo
paper_authors._id: 653f175be8ed050cb3513a36
paper_authors.name: Zheng Zhang
paper_authors._id: 653f175be8ed050cb3513a37
paper_authors.name: Shuguang Liu
paper_authors._id: 653f175be8ed050cb3513a38
paper_authors.name: Joe Chau
paper_authors._id: 653f175be8ed050cb3513a39
paper_authors.name: Han Hu
paper_authors._id: 653f175be8ed050cb3513a3a
paper_authors.name: Peng Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-31T08:23:33.140Z
paper_authors._id: 653feb7ccf1f9c88f4928910
paper_authors.avatarUrl: /avatars/23a6a6818116683ea9485e1470a0062f.svg
paper_authors.fullname: Peng Cheng
paper_authors.type: user
paper_authors.user: cp5555
paper_title: FP8-LM: Training FP8 Large Language Models
paper_summary: In this paper, we explore FP8 low-bit data formats for efficient training of
  large language models (LLMs). Our key insight is that most variables, such as
  gradients and optimizer states, in LLM training can employ low-precision data
  formats without compromising model accuracy and requiring no changes to
  hyper-parameters. Specifically, we propose a new FP8 automatic mixed-precision
  framework for training LLMs. This framework offers three levels of FP8
  utilization to streamline mixed-precision and distributed parallel training for
  LLMs. It gradually incorporates 8-bit gradients, optimizer states, and
  distributed learning in an incremental manner. Experiment results show that,
  during the training of GPT-175B model on H100 GPU platform, our FP8
  mixed-precision training framework not only achieved a remarkable 42% reduction
  in real memory usage but also ran 64% faster than the widely adopted BF16
  framework (i.e., Megatron-LM), surpassing the speed of Nvidia Transformer
  Engine by 17%. This largely reduces the training costs for large foundation
  models. Furthermore, our FP8 mixed-precision training methodology is generic.
  It can be seamlessly applied to other tasks such as LLM instruction tuning and
  reinforcement learning with human feedback, offering savings in fine-tuning
  expenses. Our FP8 low-precision training framework is open-sourced at
  {https://github.com/Azure/MS-AMP}{aka.ms/MS.AMP}.
paper_ai_keywords: FP8 low-bit data formats mixed-precision framework gradients optimizer states distributed parallel training H100 GPU BF16 framework Megatron-LM Nvidia Transformer Engine FP8 mixed-precision training framework LLM instruction tuning reinforcement learning with human feedback generic training methodology