Learning both Weights and Connections for Efficient Neural Networks
Neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems. Also, conventional networks fix the architecture before training starts; as a result, training cannot improve the architecture. To address these limitations, we describe a method to reduce the storage and computation required by neural networks by an order of magnitude without affecting their accuracy by learning only the important connections. Our method prunes redundant connections using a three-step method. First, we train the network to learn which connections are important. Next, we prune the unimportant connections. Finally, we retrain the network to fine tune the weights of the remaining connections. On the ImageNet dataset, our method reduced the number of parameters of AlexNet by a factor of 9×, from 61 million to 6.7 million, without incurring accuracy loss. Similar experiments with VGG-16 found that the total number of parameters can be reduced by 13×, from 138 million to 10.3 million, again with no loss of accuracy.
It is good that the authors study reasonably modern and well-known (AlexNet and VGG) deep nets, so that the compression ratios seem directly relevant to recent CNN work. However, the authors do not seem to compare their work to ANY of the now-large literative on related recent methods for compressing CNN and other deep nets. They dismiss OBD in Section 2 as unsuitable for "today's large scale neural networks", but that seems unfair since OBD's saliencies only require the Hessian diagonals, and several modern methods to approximate the required Hessian diagonals could be used for baselines.But even without using ODB-related methods, there are many other baselines they could of and should of reported for comparison, including methods that reduce weights to fewer bits or approximate the CNN filter matrices using cheap SVD-based compression, or ones which approximate the fully connected layers using randomized projection (e.g. FastFood methods), and so on.The authors claim in Section 2 that such work is often "orthogonal to network pruning", but that seems to miss the point: once those methods are used, is there any real advantage to the proposed method?Importantly, some of those methods are in practice much easier to realize promised speedups (e.g. doing more multiplies in parallel using weights of fewer bits) in the dominate dense computations (i.e. BLAS ops) than the proposed sparse method (see below for more on this point).      The authors are not very clear on how (and when) their approach is necessarily efficient -- Section 4.2 mentions "900k" and "700k" iterations ... relative to AlexNet "450k".Presumably this means the authors are claiming their approach requires only 2x as long to train as AlexNet?Unfortunately they are not clear because they never report wall clock in seconds.The reason that is especially critical is that they later claim great "speedups" when their nets have 5-9x few weights.However, fewer FLOPS does not directly translate to similar drops in seconds, as is well-known and carefully mentioned in much of the other work on compressing nets.Specifically, the cache-aware and multicore efficiency of modern BLAS operations (both for CPU and GPU) makes dense matrix operations very competitive to sparse operations until sparsity is much lower than 10%.So it is not surprising that the authors never report runtime seconds -- they would almost surely NOT have noticed any actual runtime speed improvements with their approach.      This paper is also unclear on several other issues, including how (and why) they decided to reduce the learning rate (by 1/100 sometimes and by 1/10 for other experiments).If this requires a lot of experimentation, then that search time needs to be included in the cost of this compression process, making this approach even less competitive.           This paper basically tries the simplest method to sparsify a net: drop small weights and retrain.It reports impressive-sounding compression ratios (e.g. 5-9x) and thus claims victory.Finding the simplest ideas that work is in general good and should be encouraged, but the actual significance of the results and carefully describing necessary conditions required to achieve them (including tweaking the learning rates) becomes critical.Too much is left out of this paper, incluidng what stopping condition they used for the retraining process (e.g. how did they properly determine that "900k iterations" was enough (and fair), in Section 4.2?).Their reporting of only FLOPS, instead of seconds as well, makes it very clear that the authors are not very aware of the rich related literature on compressing networks, and the difficulty of actually achieving significant compressions in practice (due to the efficiency of modern dense-matrix operations over sparse ones).      In short, it is good to see a paper try to get away with the simplest approach, and show an existence proof that more of the weights in some modern CNNs could be zero and still get the same accuracy.This might have more modest appplications, such as storing networks on disk (sparsely) and then running them densely (filling in zeros explicitly for the net in RAM).But the authors seem to want and try to claim more significance than that, but without comparing to any other relevant compression methods. As stated this work is too prelimary, as the authors even basically admit in Section 4.3 about VGG ("we have not yet had a chance to prune the convolutional layers".Its limitatons and tradeoffs are not discussed, nor explored in relation to any other existing methods.