title: Efficient Post-training Quantization with FP8 Formats
summary: Recent advances in deep learning methods such as LLMs and Diffusion models
  have created a need for improved quantization methods that can meet the
  computational demands of these modern architectures while maintaining accuracy.
  Towards this goal, we study the advantages of FP8 data formats for
  post-training quantization across 75 unique network architectures covering a
  wide range of tasks, including machine translation, language modeling, text
  generation, image classification, generation, and segmentation. We examine
  three different FP8 representations (E5M2, E4M3, and E3M4) to study the effects
  of varying degrees of trade-off between dynamic range and precision on model
  accuracy. Based on our extensive study, we developed a quantization workflow
  that generalizes across different network architectures. Our empirical results
  show that FP8 formats outperform INT8 in multiple aspects, including workload
  coverage (92.64% vs. 65.87%), model accuracy and suitability for a broader
  range of operations. Furthermore, our findings suggest that E4M3 is better
  suited for NLP models, whereas E3M4 performs marginally better than E4M3 on
  computer vision tasks. The code is publicly available on Intel Neural
  Compressor: https://github.com/intel/neural-compressor.
paper_authors._id: 651395c413aa0728132a885d
paper_authors.name: Haihao Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:24:51.834Z
paper_authors._id: 60ac3318e3de7c7440abb850
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60ac3318e3de7c7440abb850/DkMrPBr6Ew_dS_c9kog4e.jpeg
paper_authors.fullname: Haihao Shen
paper_authors.type: user
paper_authors.user: Haihao
paper_authors._id: 651395c413aa0728132a885e
paper_authors.name: Naveen Mellempudi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-02T21:07:11.362Z
paper_authors._id: 651b2cc82590c4c6245a1644
paper_authors.avatarUrl: /avatars/b61c64f27041531e72af8eb131999874.svg
paper_authors.fullname: Naveen Mellempudi
paper_authors.type: user
paper_authors.user: nkmellem
paper_authors._id: 651395c413aa0728132a885f
paper_authors.name: Xin He
paper_authors._id: 651395c413aa0728132a8860
paper_authors.name: Qun Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:25:29.655Z
paper_authors._id: 63079d748b054d85f299e1e4
paper_authors.avatarUrl: /avatars/d4947439475dc81f2c9e9304382b6257.svg
paper_authors.fullname: Qun Gao
paper_authors.type: user
paper_authors.user: qgao007
paper_authors._id: 651395c413aa0728132a8861
paper_authors.name: Chang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:25:47.975Z
paper_authors._id: 637b5f16a2460cde612b63eb
paper_authors.avatarUrl: /avatars/e38a35ca22da4c5ba89bb8a87482bb61.svg
paper_authors.fullname: Chang Wang
paper_authors.type: user
paper_authors.user: changwangss
paper_authors._id: 651395c413aa0728132a8862
paper_authors.name: Mengni Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:25:55.255Z
paper_authors._id: 60ac86e8233b3b757d71bf93
paper_authors.avatarUrl: /avatars/893de2472d88478c6e1b1307d2a5fdd9.svg
paper_authors.fullname: Wang, Mengni
paper_authors.type: user
paper_authors.user: MengniWang
paper_title: Efficient Post-training Quantization with FP8 Formats
paper_summary: Recent advances in deep learning methods such as LLMs and Diffusion models
  have created a need for improved quantization methods that can meet the
  computational demands of these modern architectures while maintaining accuracy.
  Towards this goal, we study the advantages of FP8 data formats for
  post-training quantization across 75 unique network architectures covering a
  wide range of tasks, including machine translation, language modeling, text
  generation, image classification, generation, and segmentation. We examine
  three different FP8 representations (E5M2, E4M3, and E3M4) to study the effects
  of varying degrees of trade-off between dynamic range and precision on model
  accuracy. Based on our extensive study, we developed a quantization workflow
  that generalizes across different network architectures. Our empirical results
  show that FP8 formats outperform INT8 in multiple aspects, including workload
  coverage (92.64% vs. 65.87%), model accuracy and suitability for a broader
  range of operations. Furthermore, our findings suggest that E4M3 is better
  suited for NLP models, whereas E3M4 performs marginally better than E4M3 on
  computer vision tasks. The code is publicly available on Intel Neural
  Compressor: https://github.com/intel/neural-compressor.
paper_ai_keywords: LLMs Diffusion models quantization methods post-training quantization FP8 data formats machine translation language modeling text generation image classification image generation segmentation E5M2 E4M3 E3M4 workload coverage model accuracy Intel Neural Compressor