Efficient Post-training Quantization with FP8 Formats
Recent advances in deep learning methods such as LLMs and Diffusion models
have created a need for improved quantization methods that can meet the
computational demands of these modern architectures while maintaining accuracy.
Towards this goal, we study the advantages of FP8 data formats for
post-training quantization across 75 unique network architectures covering a
wide range of tasks, including machine translation, language modeling, text
generation, image classification, generation, and segmentation. We examine
three different FP8 representations (E5M2, E4M3, and E3M4) to study the effects
of varying degrees of trade-off between dynamic range and precision on model
accuracy. Based on our extensive study, we developed a quantization workflow
that generalizes across different network architectures. Our empirical results
show that FP8 formats outperform INT8 in multiple aspects, including workload
coverage (92.64% vs. 65.87%), model accuracy and suitability for a broader
range of operations. Furthermore, our findings suggest that E4M3 is better
suited for NLP models, whereas E3M4 performs marginally better than E4M3 on
computer vision tasks. The code is publicly available on Intel Neural
Compressor: https://github.com/intel/neural-compressor.
651395c413aa0728132a885d
Haihao Shen
admin_assigned
2023-09-27T10:24:51.834Z
60ac3318e3de7c7440abb850
https://cdn-avatars.huggingface.co/v1/production/uploads/60ac3318e3de7c7440abb850/DkMrPBr6Ew_dS_c9kog4e.jpeg
Haihao Shen
user
Haihao
651395c413aa0728132a885e
Naveen Mellempudi
claimed_verified
2023-10-02T21:07:11.362Z
651b2cc82590c4c6245a1644
/avatars/b61c64f27041531e72af8eb131999874.svg
Naveen Mellempudi
user
nkmellem
651395c413aa0728132a885f
Xin He
651395c413aa0728132a8860
Qun Gao
admin_assigned
2023-09-27T10:25:29.655Z
63079d748b054d85f299e1e4
/avatars/d4947439475dc81f2c9e9304382b6257.svg
Qun Gao
user
qgao007
651395c413aa0728132a8861
Chang Wang
admin_assigned
2023-09-27T10:25:47.975Z
637b5f16a2460cde612b63eb
/avatars/e38a35ca22da4c5ba89bb8a87482bb61.svg
Chang Wang
user
changwangss
651395c413aa0728132a8862
Mengni Wang
admin_assigned
2023-09-27T10:25:55.255Z
60ac86e8233b3b757d71bf93
/avatars/893de2472d88478c6e1b1307d2a5fdd9.svg
Wang, Mengni
user
MengniWang
Efficient Post-training Quantization with FP8 Formats
Recent advances in deep learning methods such as LLMs and Diffusion models
have created a need for improved quantization methods that can meet the
computational demands of these modern architectures while maintaining accuracy.
Towards this goal, we study the advantages of FP8 data formats for
post-training quantization across 75 unique network architectures covering a
wide range of tasks, including machine translation, language modeling, text
generation, image classification, generation, and segmentation. We examine
three different FP8 representations (E5M2, E4M3, and E3M4) to study the effects
of varying degrees of trade-off between dynamic range and precision on model
accuracy. Based on our extensive study, we developed a quantization workflow
that generalizes across different network architectures. Our empirical results
show that FP8 formats outperform INT8 in multiple aspects, including workload
coverage (92.64% vs. 65.87%), model accuracy and suitability for a broader
range of operations. Furthermore, our findings suggest that E4M3 is better
suited for NLP models, whereas E3M4 performs marginally better than E4M3 on
computer vision tasks. The code is publicly available on Intel Neural
Compressor: https://github.com/intel/neural-compressor.
Research demonstrates that FP8 data formats improve quantization in deep learning models, outperforming INT8 in accuracy and suitability across various tasks.
LLMs Diffusion models FP8 data formats post-training quantization E5M2 E4M3 E3M4 machine translation language modeling text generation image classification generation segmentation workload coverage Intel Neural Compressor