CNNpack: Packing Convolutional Neural Networks in the Frequency Domain
Deep convolutional neural networks (CNNs) are successfully used in a number of applications. However, their storage and computational requirements have largely prevented their widespread use on mobile devices. Here we present an effective CNN compression approach in the frequency domain, which focuses not only on smaller weights but on all the weights and their underlying connections. By treating convolutional filters as images, we decompose their representations in the frequency domain as common parts (i.e., cluster centers) shared by other similar filters and their individual private parts (i.e., individual residuals). A large number of low-energy frequency coefficients in both parts can be discarded to produce high compression without significantly compromising accuracy. We relax the computational burden of convolution operations in CNNs by linearly combining the convolution responses of discrete cosine transform (DCT) bases. The compression and speed-up ratios of the proposed algorithm are thoroughly analyzed and evaluated on benchmark image datasets to demonstrate its superiority over state-of-the-art methods.
This paper presents a framework of compressing and speeding up convolutional neural networks. The convolutional kernels are transformed into frequency domain with Discrete Cosine Transform. k-means clustering, l1 shrinking, quantization and Huffman coding is performed sequentially to reduce the consumption of memory and computational complexity. Excellent experimental results are achieved on several state of the art CNN models. 
1. The experimental results are exceptionally good, significantly better than previous methods. The authors evaluated their method on Alexnet, VGG and Googlenet, which I think are sufficiently convincing.  2. Concerns about novelty: Comparing to [1], the main added part is the DCT transformation.  While speeding up CNN with frequency domain has been proposed in [2].   3. The speedup convolution section seems confusing to me.      3.1 I believe the DCT mentioned in this paper is 2-dimensional DCT, as is defined in equation (2). The complexity of 2D DCT of a d x d patch should not be O(dlog(d)). I believe O(dlog(d)) is the complexity of 1D DCT with d elements.      3.2 Line 166 to 168 seems to be misleading. The author says that convolving with the d x d DCT bases is equivalent to performing DCT direct to the H x W input data. However, it should be performing d x d DCT at each location of the H x W input data. These are two different operations.     3.3 When calculating complexity, the authors never mentioned the number of input channels. Is the number of input channels supposed to be one?    4. I suggest the authors to compare their method with [3]. DCT is a type of decomposition with fixed basis, while in [3], sparse decomposition was adopted, and the basis is optimized while fine-tuning.  5. I'm interested in the values of eta and delta for the models evaluated in the experiments. eta and delta provides the level of sparsity for the k-means centers and residuals. This can be compared with the sparsity obtained in [3].  6. The run time of proposed method is given in Table 2 of the manuscript. I'm very curious about how the authors implemented their method with GPU. According to [3], performing sparse matrix multiplication heavily suffers from the limited memory bandwidth, and GPU is well known to be very hard to optimize. Considering the proposed method is relatively complicated, I'm wondering how the authors parallelize it efficiently with CUDA and achieve runtime speedup that is similar to theoretical speedup. I suggest the authors compare their efficiency with the cudnn implementation, which represents the state-of-the-art implementation of CNN models.  [1] Han, Song, Huizi Mao, and William J. Dally. "Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding." CoRR, abs/1510.00149 2 (2015). [2] Mathieu, Michael, Mikael Henaff, and Yann LeCun. "Fast training of convolutional networks through FFTs." arXiv preprint arXiv:1312.5851 (2013). [3] Liu, Baoyuan, et al. "Sparse convolutional neural networks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015.