Network quantization is one of the most hardware friendly techniques to enable the deployment of convolutional neural networks (CNNs) on low-power mobile devices. Recent network quantization techniques quantize each weight kernel in a convolutional layer independently for higher inference accuracy, since the weight kernels in a layer exhibit different variances and hence have different amounts of redundancy. The quantization bitwidth or bit number (QBN) directly decides the inference accuracy, latency, energy and hardware overhead. To effectively reduce the redundancy and accelerate CNN inferences, various weight kernels should be quantized with different QBNs. However, prior works use only one QBN to quantize each convolutional layer or the entire CNN, because the design space of searching a QBN for each weight kernel is too large. The hand-crafted heuristic of the kernel-wise QBN search is so sophisticated that domain experts can obtain only sub-optimal results. It is difficult for even deep reinforcement learning (DRL) Deep Deterministic Policy Gradient (DDPG)-based agents to find a kernel-wise QBN configuration that can achieve reasonable inference accuracy. In this paper, we propose a hierarchical-DRL-based kernel-wise network quantization technique, AutoQ, to automatically search a QBN for each weight kernel, and choose another QBN for each activation layer. Compared to the models quantized by the state-of-the-art DRL-based schemes, on average, the same models quantized by AutoQ reduce the inference latency by 54.06%, and decrease the inference energy consumption by 50.69%, while achieving the same inference accuracy.
This paper proposes a new method for quantizing neural network weights and activations that uses deep reinforcement learning to select the appropriate bitwidth for individual kernels in each layer. The algorithm uses a reward function that weights accuracy of the quantized model with latency, energy, and FPGA area, and leverages a high level and low-level controller to create quantized models that can take into account these factors. Compared to prior approaches taht only perform layer-wise instead of kernel-wise quantization, the quantized models can achieve better performance, or latency.
While the problem of more effectively quantizing neural network weights and activations is interesting, I found this paper hard to follow and evaluate. The proposed hierarchical deep RL method mentions a number of tricks and design decisions that are not ablated, and the notation and text around the method were difficult to follow. There were also no baselines comparing to other approaches for performing kernel-wise quantization (e.g. random search, quantize based off variance, etc.). Without improved baselines and text I would recommend rejecting this paper.
Major comments:
* Based on the analysis of kernel-wise results, it seems that a very simple strategy that chooses QBNs based on weight variance could be sufficient to achieve good performance. However, there’s no comparisons to these kinds of heuristics or even to random search over QBN per-kernel. There are also no ablations that study whether the hierarchical-RL based approach beats a baseline that just directly predicts the kernel-wise QBNs independently.  Without these baselines, it’s hard to know how well AutoQ works.
* The text that details the hierarchical RL approach with multiple controllers that is core to the new AutoQ approach is extremely hard to follow, with many undefined symbols.
Minor comments:
* FIg 1: where in the neightwork do these weight kernels come from? What network/dataset?
* You repeatedly state that ML experts obtain only sub-optimal results at selecting QBNs, can you cite something for this? What if you give experts access to additiional information like visualizations of weight distributions?
* What’s the tradeoff between using different QBNs for each kernel and specifying this number? Is there additional memory overhead?
* Table 1: hard to read the exponent in kernel-wise math, add parentheses
* Please use \citep vs. \citet where appropriate
* Why not also search for kernel-specific activation quantization? 
* The notation in Fig 3 and surrounding text is really hard to follow, e.g. w_w, h_w, and indices into states (roundup could be ceil, names are confusing iRd, eRd?).
* Eqn 2 comes out of nowhere… why optimize for log(accuracy) vs accuracy? How do you choose the user-defined constants? Decay factor?
* Hardware overhead estimator: are these models accurate independent of QBN?
* Why do you have to perform Gaussian augmentation? I didn’t follow the re-labeling of transitions after Eqn. 6, is this needed?
* Define \mu when you first use it (Eqn 4?)
* Is \delta_a in implementation details the same as \sigma around Eqn 4?
* Table 3: Is there variance in these results based off fine-tuning? Could you include error bars or stderr on these estimates?
* Figure 6: are the search spaces different for these different approaches? I.e. does AutoQ perform better due to the upper/lower bounds you set on QBN?
=============
Thank you to the authors for addressing many of my concerns. The updated paper still does not present any baselines for methods that can search or learn kernel-wise quantization parameters, e.g. run random search and select the best-performing models according to your reward function. Additionally, there were no revisions made to the text to improve the presentation, and thus I continue to recommend rejection of this work.