To realize the promise of ubiquitous embedded deep network inference, it is essential to seek limits of energy and area efficiency. To this end, low-precision networks offer tremendous promise because both energy and area scale down quadratically with the reduction in precision. Here, for the first time, we demonstrate , and VGG-16bn networks on the ImageNet classification benchmark that, at 8-bit precision exceed the accuracy of the full-precision baseline networks after one epoch of finetuning, thereby leveraging the availability of pretrained models. We also demonstrate ResNet-18, ResNet-34, and ResNet-50 4-bit models that match the accuracy of the full-precision baseline networks -the highest scores to date. Surprisingly, the weights of the low-precision networks are very close (in cosine similarity) to the weights of the corresponding baseline networks, making training from scratch unnecessary. We find that gradient noise due to quantization during training increases with reduced precision, and seek ways to overcome this noise. The number of iterations required by stochastic gradient descent to achieve a given training error is related to the square of (a) the distance of the initial solution from the final plus (b) the maximum variance of the gradient estimates. By drawing inspiration from this observation, we (a) reduce solution distance by starting with pretrained fp32 precision baseline networks and fine-tuning, and (b) combat noise introduced by quantizing weights and activations during training, by using larger batches along with matched learning rate annealing. Sensitivity analysis indicates that these techniques, coupled with proper activation function range calibration, offer a promising heuristic to discover low-precision networks, if they exist, close to fp32 precision baseline networks.
This manuscript joins a crowded space of methods for low bit quantization to enable inference on more efficient hardware. In the past, these methods often were limited to 8-bit quantization, or smaller networks, or result in accuracy degradation. This paper is part of a recent crop of methods that achieve full accuracy on ResNet50 with 4-bit weights and activations. 
The method in this paper is based around a simple, yet powerful observation: Fine-tuning at low precision introduces noise in the gradient. Using the relationship between noise, batch-size and learning rate, that has recently been receiving a lot of attention in the context of large batch training, they compensate for this added noise by increasing the batch size. 
I like the simplicity and effectiveness, and believe that this method will be a useful addition to the toolbox for low-precision inference. 
Overall, the paper is well written, and the claims are well supported experimentally. Results are demonstrated on a wide range of networks, including various configurations of ResNet, DenseNet, Inception. It's not clear whether these experiments are from a single run. If they are, with sub 1% differences between methods we are getting close to the run-to-run variability, and it would be preferable to see results averaged across multiple runs. 
Ultimately, I am on the fence if this is a sufficient contribution for acceptance. In particular, this paper claims "first evidence ... matching the accuracy of full precision". While this may in a narrow technical sense be the case, PACT https://arxiv.org/abs/1805.06085 also works on ResNet50 without an accuracy drop. While this is not published work, it was rejected at ICLR last year, making it hard to recommend acceptance here. There is also work concurrently submitted to this forum (which I obviously don't expect the authors to cite or take into account, but want to mention for the sake of completeness) such as https://openreview.net/forum?id=HyfyN30qt7 which achieves the same or better results, and does not require 8-bit BN scale factors and 32-bit bias. 
This manuscript could be made stronger in multiple ways, e.g. by combining with the recently proposed clipping techniques like Choi et al. (2018), and pushing towards 2 or 3 bit training, or eliminating all larger bit-width parameters to make for easier hardware design.