Model compression can significantly reduce the computation and memory footprint of large neural networks. To achieve a good trade-off between model size and accuracy, popular compression techniques usually rely on hand-crafted heuristics and require manually setting the compression ratio of each layer. This process is typically costly and suboptimal. In this paper, we propose a Multi-Layer Pruning method (MLPrune), which is theoretically sound, and can automatically decide appropriate compression ratios for all layers. Towards this goal, we use an efficient approximation of the Hessian as our pruning criterion, based on a Kroneckerfactored Approximate Curvature method. We demonstrate the effectiveness of our method on several datasets and architectures, outperforming previous state-of-theart by a large margin. Our experiments show that we can compress AlexNet and VGG16 by 25x without loss in accuracy on ImageNet. Furthermore, our method has much fewer hyper-parameters and requires no expert knowledge.
This paper introduces an approach to pruning the parameters of a trained neural network. The idea is inspired by the Optimal Brain Surgeon method, which relies on second derivatives of the loss w.r.t. the network parameters. Here, the corresponding Hessian matrix is approximated using the Fisher information to make the algorithm scalable to very deep networks.
Strengths:
- The method does not require hyper-parameter tuning.
- The results show the good behavior of the approach.
Weaknesses:
Novelty:
- In essence, this method relies on the work of Marten & Grosse to approximate the Hessian matrix used in the Optimal Brain Surgeon strategy. This is fine, but not of great novelty.
Method:
- It is not clear to me why the notion of binary parameters gamma is necessary. Instead of varying the gammas from 1 to 0, why not directly zero out the corresponding network parameters w?
- In essence, the objective function in Eq. 5 adds an L_1 penalty on the gamma parameters, which would be related to an L_1 penalty on the ws. Note that this strategy has been employed in the past, e.g., Collins & Kohli, 2014, "Memory Bounded Deep Convolutional Networks".
- It is not clear to me how zeroing out individual parameters will truly allows one to reduce the model afterwards. In fact, one would rather want to remove entire rows or columns of the matrix W_l, which would truly correspond to a smaller model. This was what was proposed by Wen et al., NIPS 2016 and Alvarez & Salzmann, NIPS 2016, "Learning the Number of Neurons...".
- In the past, when dealing with the Hessian matrix, people have used the so-called Pearlmutter trick (Pearlmutter, Neural Computation 2014, "Fast exact multiplication by the Hessian". In fact, in this paper, the author mentions the application to the Optimal Brain Surgeon strategy. Is there a benefit of the proposed approach over this alternative strategy?
Experiments:
- While the reported compression rates are good, it is not clear to me what they mean in practice, because the proposed algorithm zeroes out individual parameters in the matrix W_l of each layer.  This does not guarantee entire channels to be removed. As such, I would not know how to make the model actually smaller in practice. It would seem relevant to show the true gains in memory usage and in inference speed (both measured on the computer, not theoretically).
Summary:
I do appreciate the fact that the proposed method does not require hyper-parameters and that it seems to yield higher compression rates than other pruning strategies that act on individual parameters. However, novelty of the approach is limited, and I am not convinced of its actual benefits in practice.