Pruning and quantization are typical approaches to reduce the computational cost of convolutional neural network (CNN) inference. Although the idea of combining both approaches seems natural, it is surprisingly difficult to determine the effects of the combination without measuring performance on the specific hardware that the user will use. This is because the benefits of pruning and quantization strongly depend on the hardware architecture where the model is executed. For example, a CPU-like architecture with no parallelization may fully exploit the reduction of computations by unstructured pruning to improve speed, but a GPUlike massive parallel architecture would not. Further, there have been emerging proposals of novel hardware architectures, such as those supporting variable bitprecision quantization. From an engineering viewpoint, optimization for each hardware architecture is useful and important in practice, but this is in essence a brute-force approach. Therefore, in this paper, we first propose a hardwareagnostic metric for measuring computational costs. Using the proposed metric, we demonstrate that Pareto-optimal performance, where the best accuracy is obtained at a given computational cost, is achieved when a slim model with fewer parameters is moderately quantized rather than a fat model with a huge number of parameters is quantized to extremely low bit precision, such as binary or ternary. Furthermore, we empirically find a possible quantitative relation between the proposed metric and the signal-to-noise ratio during stochastic gradient descent (SGD) training, by which information obtained during SGD training provides an optimal policy for quantization and pruning. We show the Pareto frontier is improved by 4× in a post-training quantization scenario based on these findings. These findings not only improve the Pareto frontier for accuracy versus computational cost, but also provide new insights into deep neural networks.
The paper proposes a new metric to evaluate both the amount of pruning and quantization. This metric is agnostic to the hardware architecture and is simply obtained by computing the Frobenius norm of some point-wise transformation of the quantized weights. They first show empirically that this Evaluation metric is correlated with the validation accuracy. Then use this metric to provide some general rules for pruning/quantizing to preserve the highest validation accuracy. Finally, they derive a strategy to perform pruning by monitory the signal to noise ratio during training and show experimentally that such method performs better than competing ones.
Pros: - Extensive experiments were performed to test the methods, and the results seem promising.
Cons: - The paper is not very clear, and the structure is somehow confusing.
- It is not easy at first to understand the experimental setup and requires to make a lot of guesses in my opinion. 
- The paper didn't motivate properly the use of a hardware-agnostic metric in the context of the quantization and pruning. Isn't the ultimate goal of pruning/quantization is to optimize the run time/energy consumption of the specific device with the least compromise on the accuracy?
I feel that the paper currently jumps between very different ideas: 
	- Evolution of the proposed metric during training: 2.3 and 2.5. While the 2.3, the take-home message is relatively clear:  the ESN is correlated with the validation accuracy, I don't fully get the point of section 2.5: It suggests that the optimizer does some sort of pruning just by choosing a higher learning rate.   
	- Finding an optimal strategy for pruning/quantizing a network: 2.4 and 2.6. Those two sections are relatively clear, although I have some questions about the experiments.
	- Developing a new strategy based on the proposed metric to quantize and prune a network in a Pareto optimal sense: This is briefly and not very well explained in section 2.7, which sends back to 2.3, but it is hard to understand how it is exactly done. It seems that section 3 provides some empirical evidence supporting this strategy, but the description of the method is hidden in the experimental details.
Some questions: 
- In figure 3, the blue dots represent validation vs ESN at each training iteration? What about the red plot, is it obtained by quantization of the parameters at different stages of training, or is it using the final parameters? Which equation was used to compute the red curve (2) or (4)?  How much quantization was performed? If the quantization was chosen to match the level of noise then it seems natural to expect such behavior in figure 3.  
- In figure 4, how much pruning was performed for each network and was it the same quantization? In other words how each point in the plot was obtained?  The authors come to the conclusion that one should 'prune until the limit of the desired accuracy and then quantize', but it is hard for me to reach the same conclusions as I don't see the separate effect of pruning and quantization in those figures. Or maybe pruning is implicitly done by choosing a small network? In this case, it makes more sense, but still, some clarifications are needed.
- Which equation for the ESN was used to produce figure 5? Equation (2) or (4)? 
- What is the Pareto frontier? I think it is worth first introducing this concept and describing more precisely how those curves are obtained. For someone who is not very familiar with these concepts, which is my case, it makes the reading very hard. 
- How was the number of pruned filters computed in figure 5 (right)? I don't expect the solutions to be sparse during training, especially that no sparsity constraint was imposed, or was it?
	
-------------------------------------------------------------------------------
Revision:
Thank you for all the clarifications and the effort to make provide a clearer version of the paper. 
Regarding section 2.7: ESNa FOR QUANTIZATION: Would it make sense to include the paragraph 2.7 at the end of 2.3, since it related to it and doesn't seem to require any of the intermediate subsections.
response to Comment 3: Unfortunately, I'm not convinced by the explanation about the effect of the lr on sparsity. The decay coefficient controls the saparsity indeed, but not the lr. That is because unlike the lr, the decay coefficient defines the cost functions to be optimized:  L+dc ||W||^2, while the lr corresponds simply to the discretization of some gradient flow.   For instance, in a deterministic and convex setting, the solution that is obtained would be the same, regardless of the chosen lr  ( provided the lr is smal enough so that the algorithm converges) see for instance [1].  In a non-convex and stochastic setting do the authors have a particular reference in mind? I'm not aware of such behavior. I would expect a similar sparsity if dc is kept fixed and only lr changed. Is it likely that with smaller lr, the algorithm just didn't have time to converge? This would explain why the obtained solutions were less sparse.
response to answer 5:  it is indeed well known that L1 norm induces sparsity, however l2 doesn't, it just encourages the weights to be smaller. In the optimization litterature sparsity of x% means x% of the parameters  are exactly 0. This is achieved with l1 norm, however l2 norm would only enforce that the coefficients are small but not necessarily 0 (see [1])
[1]:  ROBERT TIBSHIRANI, Regression Shrinkage and Selection via the Lasso.
Although the paper improved in terms of clarifications and experimental details, I still think it will benefit from additional work on careful interpretation of the results.