Training Deep Neural Networks with 8-bit Floating Point Numbers
The state-of-the-art hardware platforms for training Deep Neural Networks (DNNs) are moving from traditional single precision (32-bit) computations towards 16 bits of precision -in large part due to the high energy efficiency and smaller bit storage associated with using reduced-precision representations. However, unlike inference, training with numbers represented with less than 16 bits has been challenging due to the need to maintain fidelity of the gradient computations during back-propagation. Here we demonstrate, for the first time, the successful training of DNNs using 8-bit floating point numbers while fully maintaining the accuracy on a spectrum of Deep Learning models and datasets. In addition to reducing the data and computation precision to 8 bits, we also successfully reduce the arithmetic precision for additions (used in partial product accumulation and weight updates) from 32 bits to 16 bits through the introduction of a number of key ideas including chunk-based accumulation and floating point stochastic rounding. The use of these novel techniques lays the foundation for a new generation of hardware training platforms with the potential for 2 − 4× improved throughput over today's systems.
This manuscript provides a clear step forward in low precision training, through a mixture of novel ideas and solid experimental results. To summarize the paper: 1. Propose a new data format, or rather, recipe for combining data formats, that allows for training with a majority of the operations performed in an 8-bit floating point format. This includes some novel tricks like chunk-based accumulation 2. Show comprehensive experiments on Cifar10-ResNet and ImageNet1k-AlexNet to validate the method.  3. Demonstrate in silicon that that these methods are useful for optimized deep learning hardware.  Overall it's a solid paper, and there are lots of things I like about it. The authors clearly understand the challenges of low precision training, and are familiar with tricks of the trade such as stochastic rounding and loss scaling. Figure 1 provides a great summary of the 3 major issues that practitioners encounter in low precision training, disentangling accumulation, weight update and storage of weights, activations and gradients. It's commendable that the authors take the space to explain the basics like this even in the 8 page NIPS format. The same can be said about Figure 2, it's a great way to succinctly capture all of the operations and what precision they are in.   There are no major flaws in the manuscript, but a few minor points that could use clarification:   Why was the (1,5,2) format chosen for fp8 and (1,6,9) for fp16? The 5 exponent bits for fp8 might seem natural, since it's the same as IEEE fp16, but did the authors explore e.g. (1,4,3) or (1,6,1) and find them to work less well? If IEEE (1,5,10) does not have enough dynamic range, wouldn't (1,7,8) do even better?   Figure text is too small and unreadable in print or of PDF viewers that don't easily allow zooming in. I realize space is at a premium, but I would consider removing Figure 7 and section 4.4, as it's independent of the rest of the paper, and I am sure there is a separate publication in the works that will going into a great deal more detail on the hardware. Otherwise consider shortening the introduction, or removing some of the detail on Stochastic Rounding. Line 140 onwards provides a useful refresher, but the results in Table 4 and section 4.3 are not novel and could be removed / shortened.   Minor comments - I was not familiar with the term "Swamping", is there a reference for where this expression comes from? - 161 seems to have a stray link - 192, does ResNet18, 50 refer to the i1k sized network, or CIFAR? Please clarify. - Table 2, what does the column "FP32" mean? Is this the accuracy reported by various authors for training the same fp32 model? Should they not all be the same? - Figure 4, ResNet50 test error is around 29%, but should be closer to 24% based on He 2016. Benchmarks such as DawnBench and MLPerf generally require 25% or better on ResNet50. Can you describe differences between this ResNet and the original one and explain the discrepancy? - 253, typo, dominated -> dominant    ——-  Update after rebuttal: I don’t have much to add in response to the rebuttal, but would like to urge the authors to incorporate the suggestions from Reviewer 1, which I found very valuable.