title: Quartet: Native FP4 Training Can Be Optimal for Large Language Models
summary: The rapid advancement of large language models (LLMs) has been paralleled by
  unprecedented increases in computational demands, with training costs for
  state-of-the-art models doubling every few months. Training models directly in
  low-precision arithmetic offers a solution, by improving both computational
  throughput and energy efficiency. Specifically, NVIDIA's recent Blackwell
  architecture facilitates extremely low-precision operations, specifically FP4
  variants, promising substantial efficiency gains. Yet, current algorithms for
  training LLMs in FP4 precision face significant accuracy degradation and often
  rely on mixed-precision fallbacks. In this paper, we systematically investigate
  hardware-supported FP4 training and introduce Quartet, a new approach enabling
  accurate, end-to-end FP4 training with all the major computations (in e.g.
  linear layers) being performed in low precision. Through extensive evaluations
  on Llama-type models, we reveal a new low-precision scaling law that quantifies
  performance trade-offs across varying bit-widths and allows us to identify a
  "near-optimal" low-precision training technique in terms of
  accuracy-vs-computation, called Quartet. We implement Quartet using optimized
  CUDA kernels tailored for NVIDIA Blackwell GPUs, and show that it can achieve
  state-of-the-art accuracy for FP4 precision, successfully training
  billion-scale models. Our method demonstrates that fully FP4-based training is
  a competitive alternative to standard-precision and FP8 training. Our code is
  available at https://github.com/IST-DASLab/Quartet.
paper_authors._id: 682da9d3781210358218a950
paper_authors.name: Roberto L. Castro
paper_authors._id: 682da9d3781210358218a951
paper_authors.name: Andrei Panferov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T14:07:46.802Z
paper_authors._id: 623753b5eddd7763adc9346a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623753b5eddd7763adc9346a/rcpQAKZNrkn1-tMtraQBX.jpeg
paper_authors.fullname: Andrei Panferov
paper_authors.type: user
paper_authors.user: BlackSamorez
paper_authors._id: 682da9d3781210358218a952
paper_authors.name: Soroush Tabesh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-26T14:25:12.072Z
paper_authors._id: 632a2e325f2ff1958c0103be
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/632a2e325f2ff1958c0103be/Tb0ql9e4LcaFktTK1hzqe.jpeg
paper_authors.fullname: Rush Tabesh
paper_authors.type: user
paper_authors.user: soroushtabesh
paper_authors._id: 682da9d3781210358218a953
paper_authors.name: Oliver Sieberling
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-26T14:25:18.736Z
paper_authors._id: 659ddfb45673a33b5db22d57
paper_authors.avatarUrl: /avatars/ae1dce603b4cae2659d6070e8ce98b15.svg
paper_authors.fullname: Oliver Sieberling
paper_authors.type: user
paper_authors.user: OliverSieberling
paper_authors._id: 682da9d3781210358218a954
paper_authors.name: Jiale Chen
paper_authors._id: 682da9d3781210358218a955
paper_authors.name: Mahdi Nikdan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-26T14:25:45.476Z
paper_authors._id: 6526b8ebba9a8279c139616b
paper_authors.avatarUrl: /avatars/09f6b677603a03be128996a0765233e6.svg
paper_authors.fullname: Mahdi Nikdan
paper_authors.type: user
paper_authors.user: mnikdan97
paper_authors._id: 682da9d3781210358218a956
paper_authors.name: Saleh Ashkboos
paper_authors._id: 682da9d3781210358218a957
paper_authors.name: Dan Alistarh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-26T14:25:55.295Z
paper_authors._id: 64d100c5d8d0927372e3d4c0
paper_authors.avatarUrl: /avatars/91d9e4f1dab25b70d901783cdfcd2fd1.svg
paper_authors.fullname: Dan Alistarh
paper_authors.type: user
paper_authors.user: dalistarh
paper_title: Quartet: Native FP4 Training Can Be Optimal for Large Language Models
paper_summary: The rapid advancement of large language models (LLMs) has been paralleled by
  unprecedented increases in computational demands, with training costs for
  state-of-the-art models doubling every few months. Training models directly in
  low-precision arithmetic offers a solution, by improving both computational
  throughput and energy efficiency. Specifically, NVIDIA's recent Blackwell
  architecture facilitates extremely low-precision operations, specifically FP4
  variants, promising substantial efficiency gains. Yet, current algorithms for
  training LLMs in FP4 precision face significant accuracy degradation and often
  rely on mixed-precision fallbacks. In this paper, we systematically investigate
  hardware-supported FP4 training and introduce Quartet, a new approach enabling
  accurate, end-to-end FP4 training with all the major computations (in e.g.
  linear layers) being performed in low precision. Through extensive evaluations
  on Llama-type models, we reveal a new low-precision scaling law that quantifies
  performance trade-offs across varying bit-widths and allows us to identify a
  "near-optimal" low-precision training technique in terms of
  accuracy-vs-computation, called Quartet. We implement Quartet using optimized
  CUDA kernels tailored for NVIDIA Blackwell GPUs, and show that it can achieve
  state-of-the-art accuracy for FP4 precision, successfully training
  billion-scale models. Our method demonstrates that fully FP4-based training is
  a competitive alternative to standard-precision and FP8 training. Our code is
  available at https://github.com/IST-DASLab/Quartet.
paper_ai_summary: Quartet, a hardware-supported FP4 training approach for large language models, demonstrates state-of-the-art accuracy while significantly reducing computational costs compared to standard or FP8 precision.
paper_ai_keywords: large language models low-precision arithmetic Blackwell architecture FP4 mixed-precision linear layers low-precision scaling law CUDA kernels