title: Scalify: scale propagation for efficient low-precision LLM training
summary: Low-precision formats such as float8 have been introduced in machine learning
  accelerated hardware to improve computational efficiency for large language
  models training and inference. Nevertheless, adoption by the ML community has
  been slowed down by the complex, and sometimes brittle, techniques required to
  match higher precision training accuracy. In this work, we present Scalify, a
  end-to-end scale propagation paradigm for computational graphs, generalizing
  and formalizing existing tensor scaling methods. Experiment results show that
  Scalify supports out-of-the-box float8 matrix multiplication and gradients
  representation, as well as float16 optimizer state storage. Our JAX
  implementation of Scalify is open-sourced at
  https://github.com/graphcore-research/jax-scalify
paper_authors._id: 66a1b2c604ab5290ec0d9625
paper_authors.name: Paul Balança
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T09:08:50.489Z
paper_authors._id: 60d38165d7b174177faabd67
paper_authors.avatarUrl: /avatars/809b2a3fc3502eefcd48aa70559f8275.svg
paper_authors.fullname: Paul Balanca
paper_authors.type: user
paper_authors.user: balancap
paper_authors._id: 66a1b2c604ab5290ec0d9626
paper_authors.name: Sam Hosegood
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T09:08:56.499Z
paper_authors._id: 65776369bc075aaecd83c517
paper_authors.avatarUrl: /avatars/ff9e69a9a8a4043fd761ee69da80def9.svg
paper_authors.fullname: Sam Hosegood
paper_authors.type: user
paper_authors.user: slhosegood
paper_authors._id: 66a1b2c604ab5290ec0d9627
paper_authors.name: Carlo Luschi
paper_authors._id: 66a1b2c604ab5290ec0d9628
paper_authors.name: Andrew Fitzgibbon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T09:09:09.934Z
paper_authors._id: 6360088814657fb8cff4a88a
paper_authors.avatarUrl: /avatars/abc7f05b6c2d1bd97a1d6d2390209601.svg
paper_authors.fullname: Andrew Fitzgibbon
paper_authors.type: user
paper_authors.user: awf
paper_title: Scalify: scale propagation for efficient low-precision LLM training
paper_summary: Low-precision formats such as float8 have been introduced in machine learning
  accelerated hardware to improve computational efficiency for large language
  models training and inference. Nevertheless, adoption by the ML community has
  been slowed down by the complex, and sometimes brittle, techniques required to
  match higher precision training accuracy. In this work, we present Scalify, a
  end-to-end scale propagation paradigm for computational graphs, generalizing
  and formalizing existing tensor scaling methods. Experiment results show that
  Scalify supports out-of-the-box float8 matrix multiplication and gradients
  representation, as well as float16 optimizer state storage. Our JAX
  implementation of Scalify is open-sourced at
  https://github.com/graphcore-research/jax-scalify
paper_ai_keywords: low-precision formats float8 machine learning accelerated hardware computational efficiency large language models training and inference tensor scaling methods computational graphs float8 matrix multiplication float16 optimizer state storage JAX