PER-TENSOR FIXED-POINT QUANTIZATION OF THE BACK-PROPAGATION ALGORITHM
The high computational and parameter complexity of neural networks makes their training very slow and difficult to deploy on energy and storage-constrained computing systems. Many network complexity reduction techniques have been proposed including fixed-point implementation. However, a systematic approach for designing full fixed-point training and inference of deep neural networks remains elusive. We describe a precision assignment methodology for neural network training in which all network parameters, i.e., activations and weights in the feedforward path, gradients and weight accumulators in the feedback path, are assigned close to minimal precision. The precision assignment is derived analytically and enables tracking the convergence behavior of the full precision training, known to converge a priori. Thus, our work leads to a systematic methodology of determining suitable precision for fixed-point training. The near optimality (minimality) of the resulting precision assignment is validated empirically for four networks on the CIFAR-10, CIFAR-100, and SVHN datasets. The complexity reduction arising from our approach is compared with other fixed-point neural network designs.
This paper proposes a FX (fixed point) framework to calculate the reduced bit numbers, which can (I) use float numbers with the reduced bits to represent each NN layer's weight values W_l, activation values A_l, gradients of weights G_l^W, gradients of nodes G_{l+1}^A, and the cumulated (updated) weight W_l^(acc); (II) with the reduced representations, the training loss and testing loss will not be sacrificed much comparing with the original FL framework, where each float number is 32-bit.
Some positive points:
(a) The proposed FX framework can reduce cost for both inference and training.
(b) The experimental results looks promising.
(c) The Criteria 1-5 seems systematic and the conditions in Claim 1 can be used to calculate the required bit numbers. The author proposed an implementation of Claim 1.
Some negative points / questions:
(a) The most important part of the paper is Criteria 1-5. Criteria 1 generalizes the idea from Sakr et al. 2017, to force the contributions of weight and activation almost at the same order, which seems reasonable to make the mismatch budget p_m smallest. But the other criteria (with their corresponding notions, e.g., Criterion 2 and the concept of clipping rate \beta) are introduced in a way which is not clear enough and make the audience confused. For example, why is clipping rate \beta and relative quantization bias \eta is needed here and what is their relationship with the usual weight gradient clip norm (5% target \beta and 1% \eta target correspond to what order of clip norm)? Criteria 4 & 5 are introduced in the same way with one sentence explained like heuristics. They seem to me are introduced just for reducing corresponding bit numbers. More motivation and explanation of introducing these criteria and notions are needed.
(b) For W_l and A_l, the necessary bit numbers are calculated using Criteria 1 & EFQN condition. But for gradients, their PDRs and Deltas are calculated using other criteria & conditions. Then how to calculate their bit numbers from PDRs and Deltas?
(c) The proof of Lemma 2 & 3 directly used CLT for mini-batch average gradient items. CLT is for asymptotic case and in finite sample case it is not true. So it is heuristic calculation rather than lemma with proof (If seeking proof then some finite-sample argument like Berry-Esseen theorem is needed to quantify the probability of the average is not Gaussian). And what is the mini-batch size used here? If it is too small then probably the error of taking the mini-batch SGD as Gaussian will be large.
(d) The importance/minimality of each individual bit number in C_o is not investigated, and the claim of the near minimality of C_o cannot hold according to the current experiments. The experiments of C_{+1} and C_{-1} do not exclude the possibility that some items (not the whole C_o) are minimal. More experiments (changing one or more items while fixing the others) are needed to show every item in C_o is minimal (or not). And which one of them is the most important for training/testing (how sensitive the training/testing performance is to each bit number)? Also in C_{-1} and C_{+1}, are target \beta, \eta changed? What are these changed values?
(e) From the Claim 1, it seems that these bit numbers are sufficient, and smaller than necessary to get the same training/testing performance (e.g., from the proof of Lemma 3, with the result \eta = 0.4% < 1%). So one question is what kind of \beta and \eta target are necessary for preserving the performance and what corresponding bit numbers are necessary to achieve the necessary \beta and \eta values?
(f) The computational cost definition looks different with Eq (3) in Sakr et al. 2017. Why?
Some typos:
1.Figure 2(a), B_{A_l} is 8 for Layer 1, but 9 in Appendix Table.
2. In the last third paragraph of Page 7, is it 2.6 = (148/56.5) instead of 2.6 * (148/56.5)? Same for the following numbers.
========================================
Revision: I have read the reply from the authors and it clarified several matters. I adjusted my rating of this paper (from 6 to 7).