ACCUMULATION BIT-WIDTH SCALING FOR ULTRA- LOW PRECISION TRAINING OF DEEP NETWORKS
Efforts to reduce the numerical precision of computations in deep learning training have yielded systems that aggressively quantize weights and activations, yet employ wide high-precision accumulators for partial sums in inner-product operations to preserve the quality of convergence. The absence of any framework to analyze the precision requirements of partial sum accumulations results in conservative design choices. This imposes an upper-bound on the reduction of complexity of multiply-accumulate units. We present a statistical approach to analyze the impact of reduced accumulation precision on deep learning training. Observing that a bad choice for accumulation precision results in loss of information that manifests itself as a reduction in variance in an ensemble of partial sums, we derive a set of equations that relate this variance to the length of accumulation and the minimum number of bits needed for accumulation. We apply our analysis to three benchmark networks: CIFAR-10 ResNet 32, ImageNet ResNet 18 and ImageNet AlexNet. In each case, with accumulation precision set in accordance with our proposed equations, the networks successfully converge to the single precision floating-point baseline. We also show that reducing accumulation precision further degrades the quality of the trained network, proving that our equations produce tight bounds. Overall this analysis enables precise tailoring of computation hardware to the application, yielding area-and power-optimal systems.
There has been a lot of work on limited precision training and inference for deep learning hardware, but in most of this work, the accumulators for the multiply-and-add (FMA) operations that occur for inner products are chosen conservatively or treated as having unlimited precision. The authors address this with  an analytical method to predict the number of mantissa bits needed for partial summations during the forward, delta and gradient computation ops for convolutional and fully connected layers. They propose an information theoretic approach to argue that by using fewer bits of mantissa in the accumulator than necessary, the variance of the resulting sum is less than what it would have been if sufficient bits of mantissa were used. This is surprising to me, as quantization is usually modeled as _adding_ noise, leading to an _increase_ in variance (Mc Kinstry et al. 2018), so this is a nice counterexample to that intuition. Unfortunately the result is presented in a way that implies the variance reduction is what causes the degradation in performance, while obviously (?) it's just a symptom of a deeper problem. E.g., adding noise or multiplying by a constant to get the variance to where it should be, will not help the network converge. The variance is just a proxy for lost information. The authors should make this more clear.
Loss of variance is regarded as a proxy to the error induced/loss of information due to reduced mantissa prevision. The authors present their metric called Variance Retention Ratio (VRR) as a function of the mantissa length of product terms, partial sum (accumulator) terms, and the length of the accumulation. Thereafter, the mantissa precision of the accumulator is predicted to maintain the error of accumulation within bounds by keeping the VRR as close to 1 as possible. The authors use their derived formula for VRR to predict the minimum mantissa precision needed for accumulators for three well known networks: AlexNet, ResNet 32 and ResNet 18. For tightness analysis they present convergence results while perturbing the mantissa bits to less than those predicted by their formula, and show that it leads to more than 0.5% loss in the final test error of the network.
Some questions that the manuscript leaves open in it's current form:
0. Does this analysis only apply to ReLu networks where all the accumulated terms are positive? Would a tanh nonlinearity, e.g. in an RNN, result in a different kind of swamping behavior? I don't expect the authors to add a full analysis for the RNN case if it's indeed different, but it would be nice to comment on it. 
1. Do the authors assume that the gradients and deltas will always be within the exponent range of representation? I do not find a mention of this in the paper. In other words, are techniques like loss scaling, etc. needed in addition? Other studies in literature analyzing IEEE fp16 seem to suggest so.
2. The authors do not provide details on how they actually performed the experiments when running convergence experiments. It is not straightforward to change the bit width of the accumulator mantissa in CPU or GPU kernel libraries such as CUDNN or Intel MKL. So how do they model this?
3. On page 7, the authors point out that they provide a theoretical justification of why the chunk size should neither be too small or too large - but I do not see such a justification in the paper. More detailed explanation is needed.
There are a few minor typos at a few places, e.g.
 
1. Page 4: “… , there is a an accumulation length….”
2. Page 6: “…floaintg-point format…"
Some figures, notably 2 and 5, use text that is unreadably small in the captions. I know this is becoming somewhat common practice in conference submissions with strict pages limits, but I implore the authors to consider shaving off space somewhere else. Some of us still read on paper, or don't have the best eyes!