Learning Structured Sparsity in Deep Neural Networks
High demand for computation resources severely hinders deployment of large-scale Deep Neural Networks (DNN) in resource constrained devices. In this work, we propose a Structured Sparsity Learning (SSL) method to regularize the structures (i.e., filters, channels, filter shapes, and layer depth) of DNNs. SSL can: (1) learn a compact structure from a bigger DNN to reduce computation cost; (2) obtain a hardware-friendly structured sparsity of DNN to efficiently accelerate the DNN's evaluation. Experimental results show that SSL achieves on average 5.1× and 3.1× speedups of convolutional layer computation of AlexNet against CPU and GPU, respectively, with off-the-shelf libraries. These speedups are about twice speedups of non-structured sparsity; (3) regularize the DNN structure to improve classification accuracy. The results show that for CIFAR-10, regularization on layer depth reduces a 20-layer Deep Residual Network (ResNet) to 18 layers while improves the accuracy from 91.25% to 92.60%, which is still higher than that of original ResNet with 32 layers. For AlexNet, SSL reduces the error by ∼ 1%.
This paper proposes using group sparsity on various CNN parameters to speed up their execution. The CNN is pre-trained using a classical baseline, then starting from the corresponding parameters, group sparsity constraints are applied during retraining. The inactive groups are then removed, and the final network is fine tuned without the sparsity constraints. This simple method gives good speed-ups without much loss in accuracy (and even some time improvements).
Using group sparsity to turn off redundant parts of a CNN and improve its speed seems like a good idea. Indeed,  significant speed-ups are obtained in a large variety of experiments, with little loss in accuracy and even sometimes a small improvement. The authors use group sparsity on several axes, including the number of filters and channels used, the shape of the filters (I didn't really understand how the authors deactivate efficiently certain filters sites, this should be clarified). and the number of layers (using shortcuts to prevent the network from being disconnected). The idea explored in the paper is thus rather straightforward, but it is a good  and probably useful one. However, unless I missed something, there are many details missing: How is the group sparsity optimisation performed within the CNN training? As importantly, how are the regularization weights chosen? Are they cross-validated? For example, Table 2 reports results with "different strengths of structured sparsity regularisation". These missing details are important, because in the end there must be a balance between speed and accuracy. If the network is to be used in practice it is not enough to duplicate baseline results on test data. The authors should clarify these points.