Dynamic Network Surgery for Efficient DNNs
Deep learning has become a ubiquitous technology to improve machine intelligence. However, most of the existing deep models are structurally very complex, making them difficult to be deployed on the mobile platforms with limited computational power. In this paper, we propose a novel network compression method called dynamic network surgery, which can remarkably reduce the network complexity by making on-the-fly connection pruning. Unlike the previous methods which accomplish this task in a greedy way, we properly incorporate connection splicing into the whole process to avoid incorrect pruning and make it as a continual network maintenance. The effectiveness of our method is proved with experiments. Without any accuracy loss, our method can efficiently compress the number of parameters in LeNet-5 and AlexNet by a factor of 108× and 17.7× respectively, proving that it outperforms the recent pruning method by considerable margins. Code and some models are available at https://github.com/yiwenguo/Dynamic-Network-Surgery.
damage. Since the pruned connections have no chance to come back, incorrect pruning may cause severe accuracy loss. In consequence, the compression rate must be over suppressed to avoid such loss. Another issue is learning inefficiency. As in the paper [9], several iterations of alternate pruning and retraining are necessary to get a fair compression rate on AlexNet, while each retraining process consists of millions of iterations, which can be very time consuming.
In this paper, we attempt to address these issues and pursue the compression limit of the pruning method. To be more specific, we propose to sever redundant connections by means of continual network maintenance, which we call dynamic network surgery. The proposed method involves two key operations: pruning and splicing, conducted with two different purposes. Apparently, the pruning operation is made to compress network models, but over pruning or incorrect pruning should be responsible for the accuracy loss. In order to compensate the unexpected loss, we properly incorporate the splicing operation into network surgery, and thus enabling connection recovery once the pruned connections are found to be important any time. These two operations are integrated together by updating parameter importance whenever necessary, making our method dynamic.
In fact, the above strategies help to make the whole process flexible. They are beneficial not only to better approach the compression limit, but also to improve the learning efficiency, which will be validated in Section 4. In our method, pruning and splicing naturally constitute a circular procedure and dynamically divide the network connections into two categories, akin to the synthesis of excitatory and inhibitory neurotransmitter in human nervous systems [17].
The rest of this paper is structured as follows. In Section 2, we introduce the related methods of DNN compression by briefly discussing their merits and demerits. In Section 3, we highlight our intuition of dynamic network surgery and introduce its implementation details. Section 4 experimentally analyses our method and Section 5 draws the conclusions. The pipeline of (a) our dynamic network surgery and (b) Han et al.'s method [9], using AlexNet as an example. [9] needs more than 4800K iterations to get a fair compression rate (9×), while our method runs only 700K iterations to yield a significantly better result (17.7×) with comparable prediction accuracy.
This paper proposes a method for DNN compression which is essential for reducing the storage and processing power requirements of very deep networks.  The state-of-the-art method on deep network compression uses a variety of hashing and quantization mechanisms followed by a network pruning and retraining phase. In this paper, the authors propose a method for network pruning and retraining phase of the deep network compression method of Han et al. by introducing the option of restoring previously pruned connections. They show the efficiency of their method through experimental evaluation on several benchmark datasets.  Reported results suggest that their method performs better in terms of compression rate vs. accuracy trade-off in most of the benchmarks.
I think having the option of restoring the pruned connections is a very good idea and apparently it works well on the experimented benchmarks.  On the other hand, the argument that Han et al's method can cause permanent damage in the network does not seem exactly accurate. Because, actually their network performs better than yours in  LeNet-300-100 Ref vs LeNet-300-100 Pruned (with 0.05 loss whereas your network has 0.29 loss). This is also the case for AlexNet. Their accuracy gap is better than yours.  So, I don't think there is a permanent damage issue there. Secondly, they make multiple sessions of pruning and retraining so the network always has the opportunity to compensate for the erroneous pruning.  Your method is better in terms of run time because it requires less iterations, yet, it also brings additional space requirements that do not exist in Han et al.'s method. I think a small discussion on that issue would be great. You report results on all the benchmark datasets Han et al. used except for VGG-16. I wonder what is the reason for skipping VGG-16, which is the deepest and largest network.   In Line 100, you state "For the convolutional layers with learnable kernels, we unfold the coefficients of each kernel into a vector and concatenate all of them to Wk as a matrix." What is the reason for this?  I think the manuscript lacks bunch of details about the parameter selection which might be very relevant to the audience. In Algorithm 1, the parameter \alpha is used for updating the learning \beta. This parameter is not explained. Which learning rate update scheme do you use? For updating the parameters, you state that you have tested several candidates and decided upon magnitude-based elimination. Can you elaborate on what other schemes you have tested? On line 146, you say "the absolute value of the input". I think it should be the absolute value of the weight or parameter. The input is not the same as weights. How you decide on thresholds for parameter importance is left unclear. Can you provide more details on how you select a_k, b_k and t? Also, what scheme do you use to decay the update probability \sigma(iter) On line 188, you say three-layer network but according to your definition this is a two layer network. In Han et al. paper, they change the dropout probability during compression. How do you deal with dropout in your method? In Section 4.1, there is a toy example that depicts how your method works. It'd be great to see how Han et al.'s method would do on this example.  Minor comments: The figures look really nice in the paper but they are unreadable when printed BW.