Discrimination-aware Channel Pruning for Deep Neural Networks
Channel pruning is one of the predominant approaches for deep model compression. Existing pruning methods either train from scratch with sparsity constraints on channels, or minimize the reconstruction error between the pre-trained feature maps and the compressed ones. Both strategies suffer from some limitations: the former kind is computationally expensive and difficult to converge, whilst the latter kind optimizes the reconstruction error but ignores the discriminative power of channels. In this paper, we investigate a simple-yet-effective method called discrimination-aware channel pruning (DCP) to choose those channels that really contribute to discriminative power. To this end, we introduce additional discrimination-aware losses into the network to increase the discriminative power of intermediate layers and then select the most discriminative channels for each layer by considering the additional loss and the reconstruction error. Last, we propose a greedy algorithm to conduct channel selection and parameter optimization in an iterative way. Extensive experiments demonstrate the effectiveness of our method. For example, on ILSVRC-12, our pruned ResNet-50 with 30% reduction of channels outperforms the baseline model by 0.39% in top-1 accuracy.
The paper presents an approach for channel pruning in deep neural networks. The authors propose to prune channels based on their "discriminative power" -- this is achieved by introducing additional per-layer losses, and performing channel selection and network parameter optimization in an iterative fashion. Experimental results on 3 datasets (CIFAR-10, ILSVRC-12, LFW) show that the method compares favorably with respect to the state of the art.  Overall, the paper is well written, and explanations are clear. The Supp. Material provides many further details / experiments. However, some parts of the text could be made clearer (e.g. : lines 47-53 pg. 2 became clear to me only after reading Sec. 3; notation in lines 135-140 pg. 4 could be simplified).  The experimental results show the effectiveness of the proposed approach, which seems to bring an original contribution. However, for some technical choices, it would be useful to get a better intuition/motivation. Did the authors experiment with different losses than (5)? Could it happen that some layers capture high-level features, which are difficult to evaluate based on the cross-entropy loss? It would be interesting to give insights about how the approach could be adapted to work for models trained with different (final) losses. Furthermore, it is unclear to me how the layers L_p (the ones with added losses) are selected. Is there any heuristic to decide how many per-layer losses should be added, and to which layers?  Finally, it would be useful to discuss how much computational effort/time the approach adds at training time.  Other notes / questions: - It would be interesting to discuss in more detail the reasons for the improvement observed on ILSVRC-12 after pruning. In some cases, did the authors observe better generalization capabilities for the model?  - It would be interesting to discuss if there are architectures for which the proposed approach is more/less suitable. - Does the quality of the pre-trained model heavily affect performances? - Sec. 3.2: When discussing limitations of reconstruction methods, it would be useful to add references supporting the claims. - When using acronyms, it would be better to write also the entire name, at the first occurrence (e.g. Stochastic Gradient Descent - SGD). - References: [10] appeared at ICCV, not CVPR. - Line 197, pg. 6: < instead of > ?