Deep convolutional neural networks (CNNs) are deployed in various applications but demand immense computational requirements. Pruning techniques and Winograd convolution are two typical methods to reduce the CNN computation. However, they cannot be directly combined because Winograd transformation fills in the sparsity resulting from pruning. Li et al. (2017) propose sparse Winograd convolution in which weights are directly pruned in the Winograd domain, but this technique is not very practical because Winograd-domain retraining requires low learning rates and hence significantly longer training time. Besides,  Liu et al. (2018)  move the ReLU function into the Winograd domain, which can help increase the weight sparsity but requires changes in the network structure. To achieve a high Winograd-domain weight sparsity without changing network structures, we propose a new pruning method, spatial-Winograd pruning. As the first step, spatial-domain weights are pruned in a structured way, which efficiently transfers the spatial-domain sparsity into the Winograd domain and avoids Winograd-domain retraining. For the next step, we also perform pruning and retraining directly in the Winograd domain but propose to use an importance factor matrix to adjust weight importance and weight gradients. This adjustment makes it possible to effectively retrain the pruned Winograd-domain network without changing the network structure. For the three models on the datasets of CIFAR-10, CIFAR-100, and ImageNet, our proposed method can achieve the Winograddomain sparsities of 63%, 50%, and 74%, respectively.
Review of Spatial-Winograd Pruning Enabling Sparse Winograd Convolution
Summary:
In this submission, the authors propose a new method for pruning weights in the presence in CNNs in which the convolution is expressed as a CNN. The goal of the project is to demonstrate that they can achieve a network which contains fewer weights (and runs faster) then the original network with minimal sacrifice in network performance and without altering the network architecture.
Major Comments:
My largest comments and concerns revolve around the degree to which the proposed pruning methods results in a model that is applicable for real world devices.
1. At the minimum, the authors should provide a table with the number of parameters in the (a) original networks, (b) network with baseline pruning method and (c) network with their pruning methods. (Are the savings entirely in convolutional filters or are there savings from fully connected layers?)
2. If the authors are indeed arguing that a goal of network pruning is to perform faster inference, then results must be shown to justify this -- as it is not obvious that speed-ups could be achieved by just pruning weights. In the case of other methods, speed-ups may be achieved by selectively pruning channel filters (as opposed to spatial positions in the convolutional filter) or pruning fully connected layers (in fact, for VGG and AlexNet, a majority of the reduction in parameters due to pruning were found in these layers).
3. Are these levels of sparsity useable in a real-world system? Often the degree of sparsity in "vanilla" CNNs are at levels (e.g. in AlexNet, 30-50% sparsity in higher layers) not high enough to be harnessed for a fast implementation. Selectively zero-ing out individual spatial components of a filter might reduce the parameter count but not achieve any speed up gains. That is, the sparsity must be structured in such a way as to permit a faster implementation [e.g. 1]
4. Considering that one of the baselines changes the network architecture itself [2], I would be curious to understand how effective this method is versus other, simple baselines that change the network architecture such as (1) decreasing the number of filter, (2) decreasing the kernel sizes, (3) swapping a convolutional filter for a separable convolution [3, 4]. All of these baselines are simple to train and experiment with and a practitioner would probably consider in many situations where speed or # parameters are a constraint.
Minor Comments:
- It is unclear from the presentation whether both proposed pruning methods may be trained in tandem or in series. Please clarify in the manuscript.
- Why does Figure 3a, 3b focus on maintaining 20% sparsity on the 1st layer of the network systematically all other layers in sparsity? What is special about the first layer?
- Why do the authors explore a different range of sparsities in Figure 3a and Figure 3b?
- The authors should discuss the source of the variability (and non-monotonicity) in the plots in Figure 3 and 4 for their proposed method. How are we to interpret this? Naively, it would be appear that the method is somewhat unpredictable in performance across a range of sparsity.
- Why do the blue and purple curves in Figure 4 not space the entire range of sparsities?
- Figure 5b. What is the relative accuracy measured with respect to? The baseline model at that particular epoch or at final asymptotic performance?
[1] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
https://openreview.net/forum?id=B1ckMDqlg
[2] Efficient sparse-winograd convolutional neural networks.
Xingyu Liu, Jeff Pool, Song Han, and William J Dally.
[3] MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, Hartwig Adam
https://arxiv.org/abs/1704.04861
[4] Xception: Deep Learning with Depthwise Separable Convolutions
François Chollet
https://arxiv.org/abs/1610.02357