Published as a conference paper at ICLR 2018 RETHINKING THE SMALLER-NORM-LESS- INFORMATIVE ASSUMPTION IN CHANNEL PRUNING OF CONVOLUTION LAYERS
Model pruning has become a useful technique that improves the computational efficiency of deep learning, making it possible to deploy solutions in resourcelimited scenarios. A widely-used practice in relevant work assumes that a smallernorm parameter or feature plays a less informative role at the inference time. In this paper, we propose a channel pruning technique for accelerating the computations of deep convolutional neural networks (CNNs) that does not critically rely on this assumption. Instead, it focuses on direct simplification of the channel-tochannel computation graph of a CNN without the need of performing a computationally difficult and not-always-useful task of making high-dimensional tensors of CNN structured sparse. Our approach takes two stages: first to adopt an end-toend stochastic training method that eventually forces the outputs of some channels to be constant, and then to prune those constant channels from the original neural network by adjusting the biases of their impacting layers such that the resulting compact model can be quickly fine-tuned. Our approach is mathematically appealing from an optimization perspective and easy to reproduce. We experimented our approach through several image learning benchmarks and demonstrate its interesting aspects and competitive performance.
In this paper, the authors propose a data-dependent channel pruning approach to simplify CNNs with batch-normalizations. The authors view CNNs as a network flow of information and applies sparsity regularization on the batch-normalization scaling parameter \gamma which is seen as a “gate” to the information flow. Specifically, the approach uses iterative soft-thresholding algorithm step to induce sparsity in \gamma during the overall training phase of the CNN (with additional rescaling to improve efficiency. In the experiments section, the authors apply their pruning approach on a few representative problems and networks. 
The concept of applying sparsity on \gamma to prune channels is an interesting one, compared to the usual approaches of sparsity on weights. However, the ISTA, which is equivalent to L1 penalty on \gamma is in spirit same as “smaller-norm-less-informative” assumption. Hence, the title seems a bit misleading. 
The quality and clarity of the paper can be improved in some sections. Some specific comments by section:
3. Rethinking Assumptions:
-	While both issues outlined here are true in general, the specific examples are either artificial or can be resolved fairly easily. For example: L-1 norm penalties only applied on alternate layers is artificial and applying the penalties on all Ws would fix the issue in this case. Also, the scaling issue of W can be resolved by setting the norm of W to 1, as shown in He et. al., 2017. Can the authors provide better examples here?
-	Can the authors add specific citations of the existing works which claim to use Lasso, group Lasso, thresholding to enforce parameter sparsity?
4. Channel Pruning
-	The notation can be improved by defining or replacing “sum_reduced”
-	ISTA – is only an algorithm, the basic assumption is still L1 -> sparsity or smaller-norm-less-informative. Can the authors address the earlier comment about “a theoretical gap questioning existing sparsity inducing formulation and actual computational algorithms”?
-	Can the authors address the earlier comment on “how to set thresholds for weights across different layers”, by providing motivation for choice of penalty for each layer? 
-	Can the authors address the earlier comment on how their approach provides “guarantees for preserving neural net functionality approximately”?
5. Experiments
-	CIFAR-10: Since there is loss of accuracy with channel pruning, it would be useful to compare accuracy of a pruned model with other simpler models with similar param.size? (like pruned-resnet-101 vs. resnet-50 in ISLVRC subsection)
-	ISLVRC: The comparisons between similar param-size models is exteremely useful in highlighting the contribution of this. However, resnet-34/50/101 top-1 error rates from Table 3/4 in (He et.al. 2016) seem to be lower than reported in table 3 here. Can the authors clarify?
-	Fore/Background: Can the authors add citations for datasets, metrics for this problem?
Overall, the channel pruning with sparse \gammas is an interesting concept and the numerical results seem promising. The authors have started with right motivation and the initial section asks the right questions, however, some of those questions are left unanswered in the subsequent work as detailed above.