Adaptive Quantization of Neural Networks
Despite the state-of-the-art accuracy of Deep Neural Networks (DNN) in various classification problems, their deployment onto resource constrained edge computing devices remains challenging due to their large size and complexity. Several recent studies have reported remarkable results in reducing this complexity through quantization of DNN models. However, these studies usually do not consider the changes in the loss function when performing quantization, nor do they take the different importances of DNN model parameters to the accuracy into account. We address these issues in this paper by proposing a new method, called adaptive quantization, which simplifies a trained DNN model by finding a unique, optimal precision for each network parameter such that the increase in loss is minimized. The optimization problem at the core of this method iteratively uses the loss function gradient to determine an error margin for each parameter and assigns it a precision accordingly. Since this problem uses linear functions, it is computationally cheap and, as we will show, has a closed-form approximate solution. Experiments on MNIST, CIFAR, and SVHN datasets showed that the proposed method can achieve near or better than state-of-the-art reduction in model size with similar error rates. Furthermore, it can achieve compressions close to floating-point model compression methods without loss of accuracy.
 I have read the responses to the concerns raised by all reviewers. I find the clarifications and modifications satisfying, therefore I keep my rating of the paper to above acceptance threshold. ----------------- ORIGINAL REVIEW: The paper proposes a method for quantizing neural networks that allows weights to be quantized with different precision depending on their importance, taking into account the loss. If the weights are very relevant, it assigns more bits to them, and in the other extreme it does pruning of the weights. This paper addresses a very relevant topic, because in limited resources there is a constrain in memory and computational power, which can be tackled by quantizing the weights of the network. The idea presented is an interesting extension to weight pruning with a close form approximate solution for computing the adaptive quantization of the weights. The results presented in the experimental section are promising. The quantization is quite cheap to compute and the results are similar to other state-of-the-art quantization methods. From the tables and figures, it is difficult to grasp the decrease in accuracy when using the quantized model, compared to the full precision model, and also the relative memory compression. It would be nice to have this reference in the plots of figure 3. Also, it is difficult to see the benefits in terms of memory/accuracy compromise since not all competing quantization techniques are compared for all the datasets. Another observation is that it seems from figure 2 that a lot of the weights are quantized with around 10 bits, and it is not clear how the compromise accuracy/memory can be turned to less memory, if possible. It would be interesting to know an analogy, for instance, saying that this adaptive compression in memory would be equivalent to quantizing all weights with n bits. OTHER COMMENTS: -missing references in several points of the paper. For instance, in the second paragraph of the introduction, 1st paragraph of section 2. - few typos: *psi -> .psi in section 2.3 *simply -> simplify in proof of lemma 2.2 *Delta -> .Delta in last paragraph of section 2.2 *l2 -> L_2 or l_2 in section 3.1 last paragraph.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Adaptive Quantization of Neural Networks
Despite the state-of-the-art accuracy of Deep Neural Networks (DNN) in various classification problems, their deployment onto resource constrained edge computing devices remains challenging due to their large size and complexity. Several recent studies have reported remarkable results in reducing this complexity through quantization of DNN models. However, these studies usually do not consider the changes in the loss function when performing quantization, nor do they take the different importances of DNN model parameters to the accuracy into account. We address these issues in this paper by proposing a new method, called adaptive quantization, which simplifies a trained DNN model by finding a unique, optimal precision for each network parameter such that the increase in loss is minimized. The optimization problem at the core of this method iteratively uses the loss function gradient to determine an error margin for each parameter and assigns it a precision accordingly. Since this problem uses linear functions, it is computationally cheap and, as we will show, has a closed-form approximate solution. Experiments on MNIST, CIFAR, and SVHN datasets showed that the proposed method can achieve near or better than state-of-the-art reduction in model size with similar error rates. Furthermore, it can achieve compressions close to floating-point model compression methods without loss of accuracy.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 I have read the responses to the concerns raised by all reviewers. I find the clarifications and modifications satisfying, therefore I keep my rating of the paper to above acceptance threshold. ----------------- ORIGINAL REVIEW: The paper proposes a method for quantizing neural networks that allows weights to be quantized with different precision depending on their importance, taking into account the loss. If the weights are very relevant, it assigns more bits to them, and in the other extreme it does pruning of the weights. This paper addresses a very relevant topic, because in limited resources there is a constrain in memory and computational power, which can be tackled by quantizing the weights of the network. The idea presented is an interesting extension to weight pruning with a close form approximate solution for computing the adaptive quantization of the weights. The results presented in the experimental section are promising. The quantization is quite cheap to compute and the results are similar to other state-of-the-art quantization methods. From the tables and figures, it is difficult to grasp the decrease in accuracy when using the quantized model, compared to the full precision model, and also the relative memory compression. It would be nice to have this reference in the plots of figure 3. Also, it is difficult to see the benefits in terms of memory/accuracy compromise since not all competing quantization techniques are compared for all the datasets. Another observation is that it seems from figure 2 that a lot of the weights are quantized with around 10 bits, and it is not clear how the compromise accuracy/memory can be turned to less memory, if possible. It would be interesting to know an analogy, for instance, saying that this adaptive compression in memory would be equivalent to quantizing all weights with n bits. OTHER COMMENTS: -missing references in several points of the paper. For instance, in the second paragraph of the introduction, 1st paragraph of section 2. - few typos: *psi -> .psi in section 2.3 *simply -> simplify in proof of lemma 2.2 *Delta -> .Delta in last paragraph of section 2.2 *l2 -> L_2 or l_2 in section 3.1 last paragraph.
assistant
 I have read the responses to the concerns raised by all reviewers. I find the clarifications and modifications satisfying, therefore I keep my rating of the paper to above acceptance threshold. ----------------- ORIGINAL REVIEW: The paper proposes a method for quantizing neural networks that allows weights to be quantized with different precision depending on their importance, taking into account the loss. If the weights are very relevant, it assigns more bits to them, and in the other extreme it does pruning of the weights. This paper addresses a very relevant topic, because in limited resources there is a constrain in memory and computational power, which can be tackled by quantizing the weights of the network. The idea presented is an interesting extension to weight pruning with a close form approximate solution for computing the adaptive quantization of the weights. The results presented in the experimental section are promising. The quantization is quite cheap to compute and the results are similar to other state-of-the-art quantization methods. From the tables and figures, it is difficult to grasp the decrease in accuracy when using the quantized model, compared to the full precision model, and also the relative memory compression. It would be nice to have this reference in the plots of figure 3. Also, it is difficult to see the benefits in terms of memory/accuracy compromise since not all competing quantization techniques are compared for all the datasets. Another observation is that it seems from figure 2 that a lot of the weights are quantized with around 10 bits, and it is not clear how the compromise accuracy/memory can be turned to less memory, if possible. It would be interesting to know an analogy, for instance, saying that this adaptive compression in memory would be equivalent to quantizing all weights with n bits. OTHER COMMENTS: -missing references in several points of the paper. For instance, in the second paragraph of the introduction, 1st paragraph of section 2. - few typos: *psi -> .psi in section 2.3 *simply -> simplify in proof of lemma 2.2 *Delta -> .Delta in last paragraph of section 2.2 *l2 -> L_2 or l_2 in section 3.1 last paragraph.