We introduce an efficient way to increase the accuracy of convolution neural networks (CNNs) based on high model utilization without increasing any computational complexity. The proposed sparse-complementary convolution replaces regular convolution with sparse and complementary shapes of kernels, covering the same receptive field. By the nature of deep learning, high model utilization of a CNN can be achieved with more simpler kernels rather than fewer complex kernels. This simple but insightful model reuses of recent network architectures, ResNet and DenseNet, can provide better accuracy for most classification tasks (CIFAR-10/100 and ImageNet) compared to their baseline models. By simply replacing the convolution of a CNN with our sparse-complementary convolution, at the same FLOPs and parameters, we can improve top-1 accuracy on ImageNet by 0.33% and 0.18% for ResNet-101 and ResNet-152, respectively. A similar accuracy improvement could be gained by increasing the number of layers in those networks by ∼ 1.5×.
This paper introduces a new design of kernels in convolutional neural networks. The idea is to have sparse but complementary kernels with predefined patterns, which altogether cover the same receptive field as dense kernels. Because of the sparsity of such kernels, deeper or wider networks can be designed at the same computational cost as networks with dense kernels.
Strengths:
- The complementary kernels come at no loss compare to standard ones
- The resulting wider networks can achieve better accuracies than the original ones
Weaknesses:
- The proposed patterns are clear for 3x3 kernels, but no solution is proposed for other dimensions
- The improvement over the baseline is not very impressive
- There is no comparison against other strategies, such as 1xk and kx1 kernels (e.g., Ioannou et al. 2016)
Detailed comments:
- The separation into + and x patterns is quite clear for 3x3 kernels. However, two such patterns would not be sufficient for 5x5 or 7x7 kernels. This idea would have more impact if it generalized to arbitrary kernel dimensions.
- The improvement over the original models are of the order of less than 1 percent. I understand that such improvements are not easy to achieve, but one could wonder if they are not due to the randomness of initialization/mini-batches. It would be more meaningful to report average accuracies and standard deviations over several runs of each experiment.
- Section 4.4 briefly discusses the comparison with using 3x1 and 1x3 kernels, mentioning that an empirical comparison is beyond the scope of this paper. To me, this comparison is a must. In fact, the discussion in this section is not very clear to me, as it mentions additional experiments that I could not find (maybe I misunderstood the authors). What I would like to see is the results of a model based on the method of Ioannou et al, 2016 with the same number of FLOPS.
- In Section 2, the authors review ideas of so-called random kernel sparsity. Note that the work of Wen et al., 2016, and that of Alvarez & Salzmann, NIPS 2016, do not really impose random sparsity, but rather aim to cancel out entire kernels, thus reducing the size of the model and not requiring implementation overhead. They also do not require pre-training and re-training, but just a single training procedure. Note also that these methods often tend not to decrease accuracy, but rather even increase it (by a similar magnitude to that in this paper), for a more compact model.
- In the context of random sparsity, it would be worth citing the work of Collins & Kohli, 2014, Memory Bounded Deep Convolutional Networks.
- I am not entirely convinced by the discussion of the grouped sparsity method in Section 3.1. In fact, the order of the channels is arbitrary, since the kernels are learnt. Therefore, it seems to me that they could achieve the same result. Maybe the authors can clarify this?
- Is there a particular reason why the central points appears in both complementary kernels (+ and x)?
- Why did the authors change the training procedure of ResNets slightly compared to the original paper, i.e., 50k training images instead of 45k training + 5k validation? Did the baseline (original model) reported here also use 50k? What would the results be with 45k?
- Fig. 5 is not entirely clear to me. What was the width of each layer? The original one or the modified one?
- It would be interesting to report the accuracy of a standard ResNet with 1.325*width as a comparison, as well as the runtime of such a model.
- In Table 4, I find it surprising that there is an actual speedup for the model with larger width. I would have expected the same runtime. How do the authors explain this?