Synaptic Strength For Convolutional Neural Network
Convolutional Neural Networks(CNNs) are both computation and memory intensive which hindered their deployment in mobile devices. Inspired by the relevant concept in neural science literature, we propose Synaptic Pruning: a data-driven method to prune connections between input and output feature maps with a newly proposed class of parameters called Synaptic Strength. Synaptic Strength is designed to capture the importance of a connection based on the amount of information it transports. Experiment results show the effectiveness of our approach. On CIFAR-10, we prune connections for various CNN models with up to 96% , which results in significant size reduction and computation saving. Further evaluation on ImageNet demonstrates that synaptic pruning is able to discover efficient models which is competitive to state-of-the-art compact CNNs such as MobileNet-V2 and NasNet-Mobile. Our contribution is summarized as following: (1) We introduce Synaptic Strength, a new class of parameters for CNNs to indicate the importance of each connections. (2) Our approach can prune various CNNs with high compression without compromising accuracy. (3) Further investigation shows, the proposed Synaptic Strength is a better indicator for kernel pruning compared with the previous approach in both empirical result and theoretical analysis.
Content:   This submission introduces a new framework for compressing neural networks. The main concept is to define the "synaptic strength" of the connection between an input layer and an output feature to the the product of the norms of the kernel of the input layer and the norm of the input layer. A large synaptic strength indicates that a certain input feature plays a substantial role in computing the output feature.  The synaptic strength is incorporated into the training procedure by the means of an additional penalty term that encourages a sparse distribution of synaptic strengths. After training all connections with synaptic strength smaller than some threshold are fixed to zero and the network is finetuned.  The authors apply their method to different networks on CIFAR (including VGG, ReseNet and DenseNet) and ImageNet (ResNet-50) and show good performance (in terms of accuracy and number of remaining parameters and computations). They also look into how the sparsity after training affects accuracy.   Quality:  The idea of synaptic strength appears intriguing. I like especially the idea of pruning whole connections between input channels and output layers instead of individual weights and it's very interesing to see in which layers most pruning is happening (Figure 3 left).  I have some minor issues with the derivations in section 3.2: * eq (1): shouln't the bias be b^{i+1}_k instead of b_{i+1}_c, i.e. the each output feature (k) has a certain bias, not the input channels (c). The same should be the case for the final summand b_c in eq. (3)-(5). * In eq (3)-(5): the convolution and adding of the bias should happen inside the nonlinearity f, not after applying the nonlinearity. Maybe there are just some parantheses missing? * going from eq (3) to eq (4), the factor gamma is moved from inside the nonlinearity outside of it. This works for nonlinearities like relu and prelu (as long as gamma is positive!), but it won't work for other nonlinearities like sigmoids. This condition should be clarified.  The results reported in the paper show that the method manages to decrease the number of parameters while keeping good performance. One other quite recent method that I would like to see compared to in this paper is Louizos et al, "Learning Sparse Neural Networks through L_0 Regularization", ICLR 2018 which reports a performance of less than 4% on CIFAR 10 with WideResNets (compared to 5% in this submission with ResNet).  One of the reasons why I would like to see this comparison is that Loizos et al report that besides compressing the network, their method also serves as a regularization method that can improve generalization and I'm wondering whether the same might hold for the proposed method in this submission. Table 1 suggests that might be the case for VGG and ResNet-18 on CIFAR10 but not on ImageNet (Table 2).  Also I'm wondering how strong the dependence on the regularization parameter lambda is. How where the used parameters choosen and how much do the results differ if a different parameter is used?  The bold numbers in Table 3 indicate that the ResNet-50 version of the submission has with 5.9M parameters the smallest number of remaining parameters, however, 3 out of the 4 comparison models in the table have even less parameters.  What exactly is the sparsity in Figure 4 (a) and (b)? The parameter lambda? The threshold tau? Something else? Why did the authors not finetune in the ablation study? If the difference would go away after finetuning, this would change the result of the ablation study, right?   Clarity:  There are some details missing: What was the threshold for pruning connections?  The captions of tables 2 and 4 are not indicating which dataset they are using (ImageNet if I'm not mistaken).  Please have your submission proof-read for English style and grammar issues.   Originality:  The authors make an interesting contribution to the wide field of compressing neural networks. For me the main contribution is to make the connection between a single input feature and a single output feature the target of the compression. I'm not aware that this has been done before -- but my knowledge of this field is like not complete.  The authors compare to some other compression techniques but are missing on some recent state-of-the-art methods that show really good performance (see above for one example)  Significance:  Compressing DNNs is a very important topic for bringing deep learning into everyday applications. The concept of synaptic strength is an very interesting contribution to that field. To help convincing other researchers of this method, I would like to see more comparison with state-of-the art and other potential applications of the method like regularization.  Minor comments: In the caption of Figure 1, "left" and "right" are switched.   Update after the rebuttal:  The authors addressed most of my questions and concerns. With the promised changes incorporated I think the paper will be a better contribution and therefore I am happy to increase my rating from 6 to 7.