Scalable Methods for 8-bit Training of Neural Networks
Quantized Neural Networks (QNNs) are often used to improve network efficiency during the inference phase, i.e. after the network has been trained. Extensive research in the field suggests many different quantization schemes. Still, the number of bits required, as well as the best quantization scheme, are yet unknown. Our theoretical analysis suggests that most of the training process is robust to substantial precision reduction, and points to only a few specific operations that require higher precision. Armed with this knowledge, we quantize the model parameters, activations and layer gradients to 8-bit, leaving at a higher precision only the final step in the computation of the weight gradients. Additionally, as QNNs require batch-normalization to be trained at high precision, we introduce Range BatchNormalization (BN) which has significantly higher tolerance to quantization noise and improved computational complexity. Our simulations show that Range BN is equivalent to the traditional batch norm if a precise scale adjustment, which can be approximated analytically, is applied. To the best of the authors' knowledge, this work is the first to quantize the weights, activations, as well as a substantial volume of the gradients stream, in all layers (including batch normalization) to 8-bit while showing state-of-the-art results over the ImageNet-1K dataset.
At the hightest level, the manuscript contains three contributions: 1. Range BN. This is a clever way to make BN more low precision friendly. The authors motivate it thoroughly and back it up with convincing experiments. A good contribution 2. A largely heuristically derived training scheme that uses 8-bit weights and activations, a low (8-bit) and high (16-bit) copy of the deltas and some other tricks such as stochastic rounding. They show ImageNet ResNet18 to 65% accuracy matching full precision, which is a solid experimental validation of the method. 3. A lengthy section that tries to predict if a certain quantization scheme  will be successful based on the cosine similarity between the original and the quantized weights.   Overall, it appears to be a well-written, though out paper. Using INT8 for training is very novel and useful.   For me the weak part of the paper is clearly Section 5. The hypothesis that angles are predictive of performance seems quite disconnected from the rest of the paper. I don't see evidence in the rest of the paper that this analysis provides anything predictive that could guide an experimenter. At the same time, it's not a theoretical result that is impactful on it's own.   It seems like there might be possible ways the authors could have tried to make more of this analysis, for example injecting angle noise into a network and measuring how it affects performance. But there is neither intuition nor data what a certain angle means. What does it tell us if the angle for binary is 37°, which is almost right in the middle between no noise and orthogonal 90°? It's obvious that some error (in angle or by any other measure) will degrade performance, but to make this useful, there needs to be a functional relationship between the two. The relationship in Fig 2a) is tenuous at best, with a gradual fall-off in cosine and a sharp knee in accuracy.   The derivation of the 2^M >> sqrt(ln(N)) condition is nicely done and promising, but the authors do not show that  it's useful, which they could do e.g. by predicting the required number of bits for various operations and then showing convergence at this number of bits. Instead, they just observer that using 8 bits leaves a huge amount  of headroom, rather than probing how tight this error bound is.   My second, minor concern is that clarity should be improved. It's not exactly clear to me what data formats are used for which computations. It's laudable that the authors provide code to check, but this should be clear from the paper. Is the 8-bit format an integer format or fixed point, and how are exponents / scaling factors determined? What's the 16-bit format used for bifurcated gradients, is it fp16 or int16? Am I understanding correctly that weights and activations are in 8-bit and the gradients have 8- and 16-bit copies, or is there also a higher precision copy of the weights (common in many other low precision schemes)? What exactly do you mean by fp32 updates, does this imply there is a 32 bit copy of the weights, and if yes, where does stochastic rounding come in? This is partially addressed in the last paragraph, but should come earlier, and more clearly. Consider explaining it in form of a table, or better yet a diagram showing the various tensors that go into computing and updating a layer, and what type they are in.   Misc comments: - The section around lines 79 is incorrectly claiming that BatchNorm has not successfully been applied in 16 bit. Köster et al. 2017 (Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks) have shown that a ResNet-110 with BN can be trained completely in 16 bit, with no high precision variance parameters.  - The statement in line 80 is not clear, is 42% supposed to be the overall SotA or for alexnet? It doesn't seem to be the right number for either - Can you elaborate on the serial dependence in line 140? Why does this matter? Does this assume specialized hardware where serial dependencies are more costly than raw FLOPS? - Figure 1: Caption should mention that this figure is for the ternary case - Figure 2a: Needs axis labels. bits on the horizontal and angle / accuracy on the vertical? - Fig2 caption: type, "with respect" should be "with respect to". Middle histogram in log scale -> all 3 are in log scale.  - line 238, 10.5% for Cifar10 ResNet50. This model should train to about 93% accuracy according to He 2016. Please provide details on exact model used and why it's so much worse.   - 245 typo "precising"  In conclusion, the paper is solidly borderline, by which I mean that a rebuttal that addresses these concerns would make me reconsider the rating.      -----------   Response to author rebuttal:  In response to the rebuttal and discussion with other reviewers I'd like to update my assessment, the paper should be accepted. The new experimental results make the paper much stronger, and the authors were able to clear up a lot of misunderstandings, which I hope will make it through to the final version of the paper (e.g. a figure showing the elements of a layer, indicating the data type for each operation)