Constructing Fast Network through Deconstruction of Convolution
Convolutional neural networks have achieved great success in various vision tasks; however, they incur heavy resource costs. By using deeper and wider networks, network accuracy can be improved rapidly. However, in an environment with limited resources (e.g., mobile applications), heavy networks may not be usable. This study shows that naive convolution can be deconstructed into a shift operation and pointwise convolution. To cope with various convolutions, we propose a new shift operation called active shift layer (ASL) that formulates the amount of shift as a learnable function with shift parameters. This new layer can be optimized end-to-end through backpropagation and it can provide optimal shift values. Finally, we apply this layer to a light and fast network that surpasses existing state-of-the-art networks.
In this paper, the authors propose a network compression framework by decomposing conventional convolution operator into a depth-wise learnable shifting operation, followed by 1x1 convolution. Intuitively, this work can be approximately seen as an improvement over MobileNet by replacing the depth-wise convolution into shift operations from [23]/[11]. The idea of learning continuous displacement in shift kernels also shares certain similar spirit to deformable convolution.  Pros of the paper: The addressed problem is clearly of importance to the machine learning community and practical applications. The proposed work is technically correct and is clearly presented. The whole paper is easy to follow. The presented method achieved promising, state of the art results in the experiments.  Cons of the paper: The novelty of this paper is somewhat weakened given the existence of previous related-works (ShiftNet/Active Convolution/MobileNet). In particular, the proposed work can be regarded as a combination of 1x1 convolution (can be found in MobileNet) and shift operations (similar ideas can be found in ShiftNet/Active Convolution). However, the authors provided some insights on the deconstruction of convolution in Section 3.1, which makes sense and justified the intuition of this combination.  A concern is that, since FLOP may not be an accurate indicator of the inference speed, the final speed performance can depend largely on the practical network architecture/implementation which is case-by-case. The fact that the network architectures are not unified between different comparing methods leaves the space for one to empirically pick a configuration that favors the proposed compression method. An example is what the authors mentioned in lines 188-189: "We noted that increasing depth caused an increase in inference time; therefore, a shallower network is preferred." As a result, it is not easy to find whether the inference speed gain mostly comes from the selected implementation/architecture or from the proposed method itself. (Of course, the same problem applies to previous methods as well.)  ======================================================================================== Update: I have read both the rebuttal and the other reviews. I decide to increase the score given the authors' additional results in response to R1. However, although the authors responded to my questions in the rebuttal, the answer somewhat dodged my concern without directly addressing it. In particular, the additional results from DW-ResNet proves that the proposed ASL module is better than MobileNet's depth-wise conv + 1x1 conv under current architecture. Since the authors' have specifically chosen the network architecture to be shallow and wide, whether the inference speed gain from the proposed ASL module is largely conditioned on the choice network architectures remains a question. It will be better if the authors can additionally include results on the comparison with ShiftNet/MobileNet under their proposed architectures (Say ASNet-110 vs ShiftNet-110). This will help people better understand the limits or properties of the proposed ASL module.