We present a novel method of compression of deep Convolutional Neural Networks (CNNs) by weight sharing through a new representation of convolutional filters. The proposed method reduces the number of parameters of each convolutional layer by learning a 1D vector termed Filter Summary (FS). The convolutional filters are located in FS as overlapping 1D segments, and nearby filters in FS share weights in their overlapping regions in a natural way. The resultant neural network based on such weight sharing scheme, termed Filter Summary CNNs or FSNet, has a FS in each convolution layer instead of a set of independent filters in the conventional convolution layer. FSNet has the same architecture as that of the baseline CNN to be compressed, and each convolution layer of FSNet has the same number of filters from FS as that of the basline CNN in the forward process. With compelling computational acceleration ratio, the parameter space of FSNet is much smaller than that of the baseline CNN. In addition, FSNet is quantization friendly. FSNet with weight quantization leads to even higher compression ratio without noticeable performance loss. We further propose Differentiable FSNet where the way filters share weights is learned in a differentiable and end-to-end manner. Experiments demonstrate the effectiveness of FSNet in compression of CNNs for computer vision tasks including image classification and object detection, and the effectiveness of DFSNet is evidenced by the task of Neural Architecture Search.
In this paper authors propose a novel idea (called Filter Summary, FS)  how to compress convolutional neural networks with 2D convolutions (kernels are 3D tensors, it is also applicable to the 1D convolutions). Compression of the convolution operation is done with weight sharing: unwrapped kernel (channel-major) is packed into 1D vector by having intersected segments (shared weights). 
The strong sides of the paper are the following:
- Proposed algorithm for efficient computation of convolution operation with FS (less number of multiplication operations with comparison to the standard convolution).
- Via experiments it is demonstrated that proposed approach provides compression of the model while having close to the baseline quality for image classification and object detection tasks and for small and large datasets.
- Proposed method gives better compression factor and better quality than state-ot-the-art models for classification and object detection tasks.
- It is experimentally showed that architecture search works for proposed convolution.
There are still ways to improve the paper:
- In the introduction section it is written that "FS is quantization friendly". I would say the main concern should be "FS quantization achieves the same quality as original FS while having higher compression factor". Standard convolutions are also quantization friendly, no? At the same time I don't see huge gain to do quantization of the model. Yes, to store it we need less memory, however, for the inference we still use float-point computation. How will this quantization be helpful for inference on CPU/GPU (memory/speed)?
- Did you implement CUDA kernel to have efficient FS convolution (I suspect the answer yes)? It would be interesting to see the training time per epoch/forward/backward comparison for standard CNN and FS CNN as soon as you are using the same training process (for models you trained to have an idea for practical usage). Also it is interesting to see comparison of inference time on CPU/GPU. 
- For illustrations (figs 1, 2, 3) of input and filters packing it would be very helpful to mark where are channel and spatial dimensions on the figure for simpler understanding the packing. Typos for the figures: "in Figure 2, two slices, marked in green and blue" -> "in Figure 2, two slices, marked in green and red"; "Figure 3 by dashed lines in green and blue that cross A, where the dashed line in red" -> "Figure 3 by dashed lines in green and blue that cross A, where the dashed line in green" (on Figure 3 there are blue and green lines only).
- In the paragraph where the number of elements of matrix A is specified to be computed, it would be better to have detailed explanation why we skip C_in*S_1 elements. It is not so trivial when you read the paper the first time.
- In section 2.3 for complexity of the three stages: seems the third stage doesn't have float-point computations (this is addition for S_2 locations in matrix A), however, the third stage is included into analysis of multiplication operations (this doesn't affect the final result of complexity)
- For DFSNet are filters ordered for FS F (monotonically located)? Or now they can be located independently and starting point defined by its alpha?
- Typo: page 7 "in Table 9" -> "in Table 2"
- In Table 4 it would be better to have results for FSNet-1 too (without quantization).
- In section 3.3 there is no details how the alpha parameter from DFSNet is used in DARTS.
- In Tables 10 and 11 there are linear quantization, what does it mean? what is the difference in quantization procedure with FSNet-WQ (maybe I missed something in the paper).