How can we efficiently compress Convolutional Neural Networks (CNN) while retaining their accuracy on classification tasks? A promising direction is based on depthwise separable convolution which replaces a standard convolution with a depthwise convolution and a pointwise convolution. However, previous works based on depthwise separable convolution are limited since 1) they are mostly heuristic approaches without a precise understanding of their relations to standard convolution, and 2) their accuracies do not match that of the standard convolution. In this paper, we propose FALCON, an accurate and lightweight method for compressing CNN. FALCON is derived by interpreting existing convolution methods based on depthwise separable convolution using EHP, our proposed mathematical formulation to approximate the standard convolution kernel. Such interpretation leads to developing a generalized version rank-k FALCON which further improves the accuracy while sacrificing a bit of compression and computation reduction rates. In addition, we propose FALCON-branch by fitting FALCON into the previous state-of-the-art convolution unit ShuffleUnitV2 which gives even better accuracy. Experiments show that FALCON and FALCON-branch outperform 1) existing methods based on depthwise separable convolution and 2) standard CNN models by up to 8× compression and 8× computation reduction while ensuring similar accuracy. We also demonstrate that rank-k FALCON provides even better accuracy than standard convolution in many cases, while using a smaller number of parameters and floating-point operations. However, existing approaches based on depthwise separable convolution have several crucial limitations. First, they are heuristic methods, and their relation to the standard convolution is not clearly identified. Second, due to their heuristic nature, generalizing the methods is difficult. Third, although they give reasonable compression and computation reduction, their accuracy is not sufficient compared to that of standard-convolution-based models.
In this paper, we propose FALCON, an accurate and lightweight method for compressing CNN. FALCON overcomes the limitations of the previous methods based on the depthwise separable convolution using the following two main ideas. First, we precisely define the relationship between the standard convolution and the depthwise separable convolution using EHP (Extended Hadamard 1 Under review as a conference paper at ICLR 2020 Product), which is our proposed mathematical formulation to correlate the standard convolution kernel with the depthwise convolution kernel and the pointwise convolution kernel. We then design FALCON by fine-tuning and reordering the results of EHP to improve the accuracy of convolution operations. Second, based on the precise definition, we generalize the FALCON to design rank-k FALCON, which further improves accuracy while sacrificing a bit of compression and computation reduction rates. We also propose FALCON-branch by fitting FALCON into the state-of-the-art convolution unit ShuffleUnitV2 which gives even higher accuracy. As a result, FALCON and FAL-CON-branch provide a superior accuracy compared to other methods based on depthwise separable convolution, with similar compression and computation reduction rates, and rank-k FALCON further improves accuracy, outperforming even the original convolution in many cases. Our contributions are summarized as follows:
• Generalization. We analyze and generalize depthwise separable convolution to our proposed EHP (Extended Hadamard Product) operation. This generalization enables a precise understanding of the relationship between depthwise separable convolution and standard convolution. Furthermore, with fine-tuning operations, it leads to our proposed method FALCON.
• Algorithm. We propose FALCON, a CNN compression method based on depthwise separable convolution. FALCON is carefully designed to compress CNN with little accuracy loss. We also propose rank-k FALCON to further improve the accuracy with a little sacrifice in compression and computation reduction rates. FALCON can be easily integrated into other architectures, and we propose FALCON-branch which combines FALCON with a branch architecture for a better performance. We theoretically analyze the compression and computation reduction rates of FALCON and other competitors.
• Experiments. We perform extensive experiments and show that FALCON 1) outperforms other state-of-the-art methods based on depthwise separable convolution for compressing CNN, and 2) provides up to 8× compression and computation reduction compared to the standard convolution while giving similar accuracies. Furthermore, we show that rank-k FALCON provides even better accuracy than the standard convolution in many cases while using a smaller number of parameters and floating-point operations.
The rest of this paper is organized as follows. Section 2 explains preliminaries. Section 3 describes our proposed method FALCON. Section 4 presents experimental results. After discussing related works in Section 5, we conclude in Section 6.
This paper proposed a model compression method: Falcon and rank-k Falcon. Both are used to compress CNN type of models by replacing standard convolution layer with a compact Falcon or rank-k Falcon layer to compress the model. Falcon's main idea is to decompose the traditional convolution kernel K into two smaller tensors, one is depthwise convolution kernel D and pointwise convolution kernel P. And DP will reconstruct the original kernel K. Since D+P's memory is  D*D*M+N*M which is smaller than the original size D*D*M*N, and thus when N is large, the memory saving could be large. The paper is in general in good writing and very easy to read.
Below I have several concerns/suggestions for this paper:
1: Novelty. What is the main difference between this method and all the other tensor decomposition based methods for CNN compression? There are so many tensor decomposition based methods for CNN, and seems Falcon belongs to one of them. The one (maybe) special for Falcon is that it only decomposes along one dimension. Why this method could perform better than other tensor based decomposition methods(some of them are having even smaller memory footprint as they decompose more dimensions) or Falcon could be one special case of it? 
2: In Section 3.3 and 3.4, the proposed Falcon and rank-k Falcon seems is fully recovering the original K, see equations above Theorem 2 and above Section 3.5, should it be minimizing the reconstruction error as other tensor decomposition methods? And how to find the solutions P and D from K? how the model is getting trained if using Falcon or rank-k Falcon? Do you have retrain step after the decomposition of K?
3: In the experiment, no any other standard compression techniques such as quantization, low-rank, weight-sharing, sparse, etc are compared.  This makes us curious about the benefit of the proposed methods over other methods.
In summary, I am mostly worried about the novelty of the paper, and wondering how the model is getting trained, and the comparison with other compression techniques.