title: Microscaling Data Formats for Deep Learning
summary: Narrow bit-width data formats are key to reducing the computational and
  storage costs of modern deep learning applications. This paper evaluates
  Microscaling (MX) data formats that combine a per-block scaling factor with
  narrow floating-point and integer types for individual elements.MX formats
  balance the competing needs of hardware efficiency, model accuracy, and user
  friction. Empirical results on over two dozen benchmarks demonstrate
  practicality of MX data formats as a drop-in replacement for baseline FP32 for
  AI inference and training with low user friction. We also show the first
  instance of training generative language models at sub-8-bit weights,
  activations, and gradients with minimal accuracy loss and no modifications to
  the training recipe.
paper_authors._id: 652df53842d54348c294cc6f
paper_authors.name: Bita Darvish Rouhani
paper_authors._id: 652df53842d54348c294cc70
paper_authors.name: Ritchie Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T08:30:39.468Z
paper_authors._id: 65038b7a1801ba3867c4b924
paper_authors.avatarUrl: /avatars/b9499df4974df46fb403a639b801c9b8.svg
paper_authors.fullname: Ritchie Zhao
paper_authors.type: user
paper_authors.user: rizhao
paper_authors._id: 652df53842d54348c294cc71
paper_authors.name: Ankit More
paper_authors._id: 652df53842d54348c294cc72
paper_authors.name: Mathew Hall
paper_authors._id: 652df53842d54348c294cc73
paper_authors.name: Alireza Khodamoradi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T08:31:35.965Z
paper_authors._id: 62841d4adcfdf66c168610ca
paper_authors.avatarUrl: /avatars/42e0144acb64969eac94e123c4f5fdbe.svg
paper_authors.fullname: Alireza Khodamoradi
paper_authors.type: user
paper_authors.user: arkhodamoradi
paper_authors._id: 652df53842d54348c294cc74
paper_authors.name: Summer Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T08:31:42.766Z
paper_authors._id: 6463d76038083255f6756434
paper_authors.avatarUrl: /avatars/5dfcee4812963dbff953b962a4d785fc.svg
paper_authors.fullname: Summer Deng
paper_authors.type: user
paper_authors.user: SummerDeng
paper_authors._id: 652df53842d54348c294cc75
paper_authors.name: Dhruv Choudhary
paper_authors._id: 652df53842d54348c294cc76
paper_authors.name: Marius Cornea
paper_authors._id: 652df53842d54348c294cc77
paper_authors.name: Eric Dellinger
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T08:32:05.486Z
paper_authors._id: 64bfd086623ce87c701c4851
paper_authors.avatarUrl: /avatars/a2baa56c0e6b19ef48de39d2c2216925.svg
paper_authors.fullname: Eric Dellinger
paper_authors.type: user
paper_authors.user: ericdellinger
paper_authors._id: 652df53842d54348c294cc78
paper_authors.name: Kristof Denolf
paper_authors._id: 652df53842d54348c294cc79
paper_authors.name: Stosic Dusan
paper_authors._id: 652df53842d54348c294cc7a
paper_authors.name: Venmugil Elango
paper_authors._id: 652df53842d54348c294cc7b
paper_authors.name: Maximilian Golub
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-22T22:28:34.941Z
paper_authors._id: 64dfcf93f6c2311e7ed48a74
paper_authors.avatarUrl: /avatars/61326c72f64c1195084a2613035d9b94.svg
paper_authors.fullname: Golub
paper_authors.type: user
paper_authors.user: mtg01
paper_authors._id: 652df53842d54348c294cc7c
paper_authors.name: Alexander Heinecke
paper_authors._id: 652df53842d54348c294cc7d
paper_authors.name: Phil James-Roxby
paper_authors._id: 652df53842d54348c294cc7e
paper_authors.name: Dharmesh Jani
paper_authors._id: 652df53842d54348c294cc7f
paper_authors.name: Gaurav Kolhe
paper_authors._id: 652df53842d54348c294cc80
paper_authors.name: Martin Langhammer
paper_authors._id: 652df53842d54348c294cc81
paper_authors.name: Ada Li
paper_authors._id: 652df53842d54348c294cc82
paper_authors.name: Levi Melnick
paper_authors._id: 652df53842d54348c294cc83
paper_authors.name: Maral Mesmakhosroshahi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-17T08:34:44.294Z
paper_authors._id: 64dfd286ead049c7b88822ba
paper_authors.avatarUrl: /avatars/f4dff23cb59de052ad48e8c2326c9b7c.svg
paper_authors.fullname: Maral Mesmakhosroshahi
paper_authors.type: user
paper_authors.user: mamesmak
paper_authors._id: 652df53842d54348c294cc84
paper_authors.name: Andres Rodriguez
paper_authors._id: 652df53842d54348c294cc85
paper_authors.name: Michael Schulte
paper_authors._id: 652df53842d54348c294cc86
paper_authors.name: Rasoul Shafipour
paper_authors._id: 652df53842d54348c294cc87
paper_authors.name: Lei Shao
paper_authors._id: 652df53842d54348c294cc88
paper_authors.name: Michael Siu
paper_authors._id: 652df53842d54348c294cc89
paper_authors.name: Pradeep Dubey
paper_authors._id: 652df53842d54348c294cc8a
paper_authors.name: Paulius Micikevicius
paper_authors._id: 652df53842d54348c294cc8b
paper_authors.name: Maxim Naumov
paper_authors._id: 652df53842d54348c294cc8c
paper_authors.name: Colin Verilli
paper_authors._id: 652df53842d54348c294cc8d
paper_authors.name: Ralph Wittig
paper_authors._id: 652df53842d54348c294cc8e
paper_authors.name: Eric Chung
paper_title: Microscaling Data Formats for Deep Learning
paper_summary: Narrow bit-width data formats are key to reducing the computational and
  storage costs of modern deep learning applications. This paper evaluates
  Microscaling (MX) data formats that combine a per-block scaling factor with
  narrow floating-point and integer types for individual elements.MX formats
  balance the competing needs of hardware efficiency, model accuracy, and user
  friction. Empirical results on over two dozen benchmarks demonstrate
  practicality of MX data formats as a drop-in replacement for baseline FP32 for
  AI inference and training with low user friction. We also show the first
  instance of training generative language models at sub-8-bit weights,
  activations, and gradients with minimal accuracy loss and no modifications to
  the training recipe.
paper_ai_summary: Microscaling data formats reduce computational and storage costs while maintaining model accuracy and enabling sub-8-bit training for generative language models.
paper_ai_keywords: narrow bit-width data formats Microscaling (MX) data formats per-block scaling factor narrow floating-point narrow integer types hardware efficiency user friction AI inference training generative language models