With the rapidly scaling up of deep neural networks (DNNs), extensive research studies on network model compression such as weight pruning have been performed for efficient deployment. This work aims to advance the compression beyond the weights to the activations of DNNs. We propose the Integral Pruning (IP) technique which integrates the activation pruning with the weight pruning. Through the learning on the different importance of neuron responses and connections, the generated network, namely IPnet, balances the sparsity between activations and weights and therefore further improves execution efficiency. The feasibility and effectiveness of IPnet are thoroughly evaluated through various network models with different activation functions and on different datasets. With < 0.5% disturbance on the testing accuracy, IPnet saves 71.1% ∼ 96.35% of computation cost, compared to the original dense models with up to 5.8× and 10× reductions in activation and weight numbers, respectively. The source codes are available at (omitted for blind review).
This article presents a novel approach called Integral Pruning (IP) to reduce the computation cost of Deep neural networks (DNN) by integrating activation pruning along with weight pruning. The authors show that common techniques of exclusive weight pruning does compress the model size, but increases the number of non-zero activations after ReLU. This would counteract the advantage of DNN accelerator designs (Albericio et al., 2016; Reagen et al., 2016) that leverage activation sparsity to speed up the computations. IP starts with pruning the weights using an existing technique to mask out weights under a threshold and then fine-tune the network in an iterative fashion to maintain the accuracy. After weight pruning, IP further masks out the activations with smaller magnitude to reduce the computation cost. Unlike weight pruning techniques that use static masks, the authors propose to use dynamic activation masks for activation sparsity in order to account for various patterns that are being activated in DNN for different input samples. In order to do this, the 'winner rate' measure for every layer (or for a group of layers in deep networks like ResNet32) is defined, to dynamically set the threshold for the generation of activation masks which eventually controls the amount of non-zero activation entries. The article empirically analyzes the sensitivity of activation pruning on validation data by setting different winner rates at every layer in DNN and decides upon a set of winner rates accordingly followed by an iteration of fine-tuning the network to maintain its performance. The authors show that their technique produced lower number of non-zero activations in comparison with the intrinsic sparse ReLU activations and weight pruning techniques. 
The topic of reducing network complexity for embedded implementations of DNNs is highly relevant, in particular for the ICLR community.
The IP technique yields significantly reduced number of multiply-accumulate operations (MACs) across different models like MLP-3, ConvNet-5, ResNet32 and AlexNet and on different datasets like MNIST, CIFAR10 and ImageNet. They also depicted that pruning the activations with dynamic activation masks followed by fine-tuning the network yields more sparse activations and negligible loss in accuracy when compared against using static activation masks.
    
Strengths of the paper:
- The motivation to extend compression beyond the weights to activations in order to support the DNN accelerator designs and the technical details are clearly explained. 
- The proposed technique indeed produces sparser activations than intrinsic ReLu sparse activations and can also applied to any network regardless of the choice of activation function.
- The proposed technique is evaluated across different network architectures and datasets.
- The advantage of adapting dynamic activation masks over static ones is clearly demonstrated.
 Weaknesses of the paper:
- The originality of the approach is limited because it is a relatively straightforward combination of existing techniques for weight and activation pruning.
- The "winner rate" measure is defined for every layer and should be explored over different values in order to find the equilibrium to reduce the number of non-zero activations and maintain the accuracy. This search of winner rates will become inefficient as the depth of the network increases. However, the authors used a single winner rate for a group of layers in case of ResNet-32 to reduce the exploration of search space but this choice might lead to suboptimal results.
- The authors compare the resultant number of MAC operations against numbers from the weight pruning technique. However, there also exist different works on group pruning techniques like Liu et al. (2017), Huang & Wang (2017), Ye et al. (2018) to prune entire channels / feature maps and thus yield more compact networks. Since these approaches prune the channels, they show a direct impact on the computation complexity and greatly reduce the computation time. A proper and fair comparison would be to compare the numbers of IP against such group pruning techniques. This comparison is highly important to highlight the significance of the approach on speeding up the DNNs and it is missing from the paper.
- At several locations in Section 4, e.g. Sec. 4.1, 4.3, and 4.4. there is no precise statement about the incurred accuracy loss (or no statement at all). The connection to Figures 4 and 5 is not immediately clear and should be made explicit.
		
References:
- Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming.
- Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang. Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers
- Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks.
        
Overall Evaluation:
The authors integrate activation pruning along with the weight pruning and show that the number of MAC operations are greatly reduced by their technique when compared to the numbers of weight pruning alone. 	However, I am not convinced regarding the reported number of MAC operations since the number of MAC operations of sparse weight matrices and activations would remain the same as the original models unless some of the filters/activation maps are pruned from the network.  On the other hand, comparisons against group pruning techniques are highly necessary to evaluate the potential impact of the approach on speeding up of DNNs. My preliminary rating is a weak reject but I am open to revise my rating based on the authors response to the above stated major weaknesses.
Minor comments:
- Caption of Fig. 4 should mention the task on which the results were obtained.
- There are occasional grammar errors and typos that should be corrected.