EFFICIENT SPARSE-WINOGRAD CONVOLUTIONAL NEURAL NETWORKS
Convolutional Neural Networks (CNNs) are computationally intensive, which limits their application on mobile devices. Their energy is dominated by the number of multiplies needed to perform the convolutions. Winograd's minimal filtering algorithm (Lavin, 2015) and network pruning (Han et al., 2015) can reduce the operation count, but these two methods cannot be directly combined -applying the Winograd transform fills in the sparsity in both the weights and the activations. We propose two modifications to Winograd-based CNNs to enable these methods to exploit sparsity. First, we move the ReLU operation into the Winograd domain to increase the sparsity of the transformed activations. Second, we prune the weights in the Winograd domain to exploit static weight sparsity. For models on CIFAR-10, CIFAR-100 and ImageNet datasets, our method reduces the number of multiplications by 10.4×, 6.8× and 10.8× respectively with loss of accuracy less than 0.1%, outperforming previous baselines by 2.0×-3.0×. We also show that moving ReLU to the Winograd domain allows more aggressive pruning.
Summary: 
The paper presents a modification of the Winograd convolution algorithm that enables a reduction of multiplications in a forward pass of 10.8x almost without loss of accuracy. 
This modification combines the reduction of multiplications achieved by the Winograd convolution algorithm with weight pruning in the following way:
- weights are pruned after the Winograd transformation, to prevent the transformation from filling in zeros, thus preserving weight sparsity
- the ReLU activation function associated with the previous layer is applied to the Winograd transform of the input activations, not directly to the spatial-domain activations, also yielding sparse activations
This way sparse multiplication can be performed. Because this yields a network, which is not mathematically equivalent to a vanilla or Winograd CNN, the method goes through three stages: dense training, pruning and retraining. The authors highlight that a dimension increase in weights and ReLU activations provide a more powerful representation and that stable dynamic activation densities over layer depths benefit the representational power of ReLU layers.
Review:
The paper shows good results using the proposed method and the description is easy to follow. I particularly like Figure 1. 
I only have a couple of questions/comments:
1) I’m not familiar with the term m-specific (“Matrices B, G and A are m-specific.”) and didn’t find anything that seemed related in a very quick google search. Maybe it would make sense to add at least an informal description.
2) Although small filters are the norm, you could add a note, describing up to what filter sizes this method is applicable. Or is it almost exactly the same as for general Winograd CNNs?
3) I think it would make sense to mention weight and activation quantization in the intro as well (even if you leave a combination with quantization for future work), e.g. Rastegari et al. (2016), Courbariaux et al. (2015) and Lin et al. (2015)
4) Figure 5 caption has a typo: “acrruacy”
References:
Courbariaux, Matthieu, Yoshua Bengio, and Jean-Pierre David. "Binaryconnect: Training deep neural networks with binary weights during propagations." In Advances in Neural Information Processing Systems, pp. 3123-3131. 2015.
Lin, Zhouhan, Matthieu Courbariaux, Roland Memisevic, and Yoshua Bengio. "Neural networks with few multiplications." arXiv preprint arXiv:1510.03009 (2015).
Rastegari, Mohammad, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. "Xnor-net: Imagenet classification using binary convolutional neural networks." In European Conference on Computer Vision, pp. 525-542. Springer International Publishing, 2016.