Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm
The high computational and parameter complexity of neural networks makes their training very slow and difficult to deploy on energy and storage-constrained comput- ing systems. Many network complexity reduction techniques have been proposed including fixed-point implementation. However, a systematic approach for design- ing full fixed-point training and inference of deep neural networks remains elusive. We describe a precision assignment methodology for neural network training in which all network parameters, i.e., activations and weights in the feedforward path, gradients and weight accumulators in the feedback path, are assigned close to minimal precision. The precision assignment is derived analytically and enables tracking the convergence behavior of the full precision training, known to converge a priori. Thus, our work leads to a systematic methodology of determining suit- able precision for fixed-point training. The near optimality (minimality) of the resulting precision assignment is validated empirically for four networks on the CIFAR-10, CIFAR-100, and SVHN datasets. The complexity reduction arising from our approach is compared with other fixed-point neural network designs.
 This papers introduces a quantization scheme for the back-propagation algorithm to reduce the bit size in the target neural networks. While the paper introduces one way to bring the quantization inside the training procedure and shows the tradeoff between number of bits and the accuracy, the paper is poorly written so it is hard to understand the paper*s main proposal. So I would recommend to re-organize the paper and introduce one toy example to illustrate how the proposed method works in the training time and the inference time. Currently the important part, the overall architecture, is explained in the appendix, not in the main paper. The main idea is rather simple, to introduce a quantizer in various components in the back-propagation algorithm. I think we need a clear explanation on "how to" quantize each tensor in each quantizer, instead of many obscure terms in the section 2 and 3. Also the important numbers are in the appendix C, but their meanings are hard to understand. Also in general, quantization is one way of reducing training and inference computational complexity. There are other ways of achieving the same purpose such as distillation to a smaller network (less parameters), etc, so in order to argue the computational gains over this obvious approach, we need a training time and inference time benchmark.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm
The high computational and parameter complexity of neural networks makes their training very slow and difficult to deploy on energy and storage-constrained comput- ing systems. Many network complexity reduction techniques have been proposed including fixed-point implementation. However, a systematic approach for design- ing full fixed-point training and inference of deep neural networks remains elusive. We describe a precision assignment methodology for neural network training in which all network parameters, i.e., activations and weights in the feedforward path, gradients and weight accumulators in the feedback path, are assigned close to minimal precision. The precision assignment is derived analytically and enables tracking the convergence behavior of the full precision training, known to converge a priori. Thus, our work leads to a systematic methodology of determining suit- able precision for fixed-point training. The near optimality (minimality) of the resulting precision assignment is validated empirically for four networks on the CIFAR-10, CIFAR-100, and SVHN datasets. The complexity reduction arising from our approach is compared with other fixed-point neural network designs.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This papers introduces a quantization scheme for the back-propagation algorithm to reduce the bit size in the target neural networks. While the paper introduces one way to bring the quantization inside the training procedure and shows the tradeoff between number of bits and the accuracy, the paper is poorly written so it is hard to understand the paper*s main proposal. So I would recommend to re-organize the paper and introduce one toy example to illustrate how the proposed method works in the training time and the inference time. Currently the important part, the overall architecture, is explained in the appendix, not in the main paper. The main idea is rather simple, to introduce a quantizer in various components in the back-propagation algorithm. I think we need a clear explanation on "how to" quantize each tensor in each quantizer, instead of many obscure terms in the section 2 and 3. Also the important numbers are in the appendix C, but their meanings are hard to understand. Also in general, quantization is one way of reducing training and inference computational complexity. There are other ways of achieving the same purpose such as distillation to a smaller network (less parameters), etc, so in order to argue the computational gains over this obvious approach, we need a training time and inference time benchmark.
assistant
 This papers introduces a quantization scheme for the back-propagation algorithm to reduce the bit size in the target neural networks. While the paper introduces one way to bring the quantization inside the training procedure and shows the tradeoff between number of bits and the accuracy, the paper is poorly written so it is hard to understand the paper*s main proposal. So I would recommend to re-organize the paper and introduce one toy example to illustrate how the proposed method works in the training time and the inference time. Currently the important part, the overall architecture, is explained in the appendix, not in the main paper. The main idea is rather simple, to introduce a quantizer in various components in the back-propagation algorithm. I think we need a clear explanation on "how to" quantize each tensor in each quantizer, instead of many obscure terms in the section 2 and 3. Also the important numbers are in the appendix C, but their meanings are hard to understand. Also in general, quantization is one way of reducing training and inference computational complexity. There are other ways of achieving the same purpose such as distillation to a smaller network (less parameters), etc, so in order to argue the computational gains over this obvious approach, we need a training time and inference time benchmark.