Compressing deep neural networks (DNNs) is important for real-world applications operating on resource-constrained devices. However, it is difficult to change the model size once the training is completed, which needs re-training to configure models suitable for different devices. In this paper, we propose a novel method that enables DNNs to flexibly change their size after training. We factorize the weight matrices of the DNNs via singular value decomposition (SVD) and change their ranks according to the target size. In contrast with existing methods, we introduce simple criteria that characterize the importance of each basis and layer, which enables to effectively compress the error and complexity of models as little as possible. In experiments on multiple image-classification tasks, our method exhibits favorable performance compared with other methods.
This paper proposes to reshape the weights of the layers of deep neural networks and parametrize them with a low-rank matrix decomposition (SVD). The rank is optimized using two criterion (error-based and complexity-based). Since the decomposition is applied post-hoc, the authors propose to correct the parameters of the batch norm analytically. The authors propose to jointly optimize a loss on the full network and the low-rank version. Experiments are done on CIFAR 10 an 100 with a VGG-15 and ResNet-34 architecture.                                                                                           
Some of the ideas are interesting and would be worth developping further. However, the paper in the current state cannot be accepted for the following reasons: (1) the novelty is low, this very type of decomposition is already widely studied (2) the paper is not clear as to what the contributions are, and why they are justified, theoretically or empirically, (3) the review and comparison with the state-of-the-art is lacking and (4) the experimental setup is simplistic and not convicing. (5) overall the paper is imprecise.
Main comments
* Applying SVD to the matricized weights of deep neural networks is not new. Actual contributions need to be separated from existing works.
* The related word needs to be reviewed. Many references are missing. In particular, the proposed method could be considered as a special case of tensor based methods.  
* The references that *are* listed in the related work are not properly reviewed: the authors aim to not compare with them claiming that they require re-training. Lebedev et al provide a method that works both for end-to-end training or post-hoc, by applying tensor decomposition to the trained weights. Fine-tuning is optional and done to recover performance.
* How was the complexity-based criterion obtained? Why use both M and P, since M includes P? How do the proposed criterion compare to simple measures, e.g. explained variance?
* The authors should compare with other compression techniques: layer-wise compression (e.g. Lebedev et al, Speeding-up convolutional neural networks using fine-tuned cp-decomposition, ICLR 2015) or full network compression (e.g. Kossaifi et al, T-Net: Parametrizing Fully Convolutional Nets with a Single High-Order Tensor, CVPR 2019).
Missing experiments
* The proposed learning loss needs to be compared with the original one to demonstrate any potential performance improvement. Currently, it is not clear whether it is actually helping. In other words, there should also be a comparison with \lambda = 1 or 0.
* Does the proposed loss have an effect on the selected rank? On the actual rank of the weights? On the distribution of the eigenvalues?
* The BN correction needs to be experimentally motivated: since the network is trained with a loss that incorporates the low-rank network, is that needed? Does the propose loss affect performance? How does performance change with and without that BN correction?
* Experiments on CIFAR 10-100 is not sufficient to be convincing, the authors should ideally try a more realistic, large scale dataset, e.g. ImageNet.
* VGG-15 is not convincing to show-case compression, as more than 80% of the parameters are in the fully-connected layer
* The authors assume that the SVD decomposition of the weights does no change significantly at each step: is there any empirical evidence supporting this assumption? This most likely depends on the experimental setup (batch-size, learning rate, etc).