Learning Sparse Neural Networks via Sensitivity-Driven Regularization
The ever-increasing number of parameters in deep neural networks poses challenges for memory-limited applications. Regularize-and-prune methods aim at meeting these challenges by sparsifying the network weights. In this context we quantify the output sensitivity to the parameters (i.e. their relevance to the network output) and introduce a regularization term that gradually lowers the absolute value of parameters with low sensitivity. Thus, a very large fraction of the parameters approach zero and are eventually set to zero by simple thresholding. Our method surpasses most of the recent techniques both in terms of sparsity and error rates. In some cases, the method reaches twice the sparsity obtained by other techniques at equal error rates.
The paper addresses the actual problem of structural sparsity of deep neural networks (NNs). The authors propose a general training procedure aiming for sparsity that measures relevance of a model parameter (weight) as analytical output sensitivity with respect to the particular weight. - The approach comprises a novel regularization technique that gradually lowers the absolute values of weights with low sensitivities, while weights with high sensitivities are not modified. After each training epoch, weights lower then a given threshold are zeroed out. - The authors also present a theoretical explanation and analysis of the method and its brief experimental evaluation on MNIST and ImageNet data sets.  Clarity: - The presentation is comprehensible and well-organized, the new method and experiments are described adequately. - The authors also summarize some the major recent related works. However, they omit existing works concerning regularization and pruning of neural networks using sensitivity analysis (e.g., [1,2]).  Typos: line 78 "rule.Then" (missing gap) line 224 "over [9] (... 68x compression ratio) over the nearest" ... 68x corresponds to the 5th row of Table 1, labeled by SparseVD[14] (not Han et al. [9])  Quality: - The paper has a fairly good technical quality. The explanation, derivation and theoretical analysis of the new method are detailed and comprehensible. However, I missed a greater experimental evaluation, especially concerning more complex NN-architectures.  It is unusual, that the authors first propose the adaptation rule (Eq. 9) and after that they derive the corresponding error function (eq. 15). The general shape of the penalization function (eq. 15) is therefore quite complex and, moreover, different activation functions thus lead to different error terms. Usually, the process is opposite: to design an error function first and then the corresponding adaptation rules.  Novelty and significance: The paper presents a new stochastic regularize-and-prune technique that is an interesting extension and a concurrent of weight decay. Contrary to weight decay, it doesn't reduce all of model weights but only weights with great "insensitivity", which may prevent the model from undesirable reduction of important weights.  (+) The new method overcomes the reference pruning techniques on MNIST data by reaching twice the sparsity for similar error rates, while the proposed regularization technique itself proved to improve generalization ability of the NN-model and prevent overfitting (it seems that it slightly overcomes weight-decay in this respect). (-) However, the experimental results for more complex NN-architecture (VGG-16) show only relatively small benefit over the baseline technique. Therefore, more experimental results for complex tasks would be beneficial.  Questions:  - Based on the presented experimental results, it seems that the method prunes more extensively layers closer to the input layer. Have you observed and investigated this possible property? - A further important question is, what is the computational complexity of the algorithm (e.g., when compared to weight-decay).  [1] Engelbrecht, A. and Cloete, I. A Sensitivity Analysis Algorithm for Pruning Feedforward Neural Networks. In: IEEE International Conference in Neural Networks, Washington, DC, USA. 1996. Vol. 2 of IEEE ICNN’96, pp. 1274–1277. [2] Mrazova, I., Kukacka, M. Can Deep Neural Networks Discover Meaningful Pattern Features?, Procedia Computer Science, Vol. 12, 2012, pp. 194-199.  _________________________________________________________________ The rebuttal addressed most of my points (recent works, complexity) except the point that the adaptation rules seem to be the same for all activation functions.  I think, the method is worth of publication, although the authors should consider more previous works and alter the paper by deeper experimental evaluation.