Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
We present techniques for speeding up the test-time evaluation of large convolutional networks, designed for object recognition tasks. These models deliver impressive accuracy, but each image evaluation requires millions of floating point operations, making their deployment on smartphones and Internet-scale clusters problematic. The computation is dominated by the convolution operations in the lower layers of the model. We exploit the redundancy present within the convolutional filters to derive approximations that significantly reduce the required computation. Using large state-of-the-art models, we demonstrate speedups of convolutional layers on both CPU and GPU by a factor of 2×, while keeping the accuracy within 1% of the original model.
The paper addresses the problem of speeding up the evaluation of pre-trained image classification ConvNets. To this end, a number of techniques are proposed, which are based on the tensor representation of the conv. layer weight matrix. Namely, the following techniques are considered (Sect. 3.2-3.5):1) SVD decomposition of the tensor;2) outer product decomposition of the tensor;3) monochromatic approximation of the first conv. layer - projecting RGB colors to a 1-D space, followed by clustering;4) biclustering tensor approximation - clustering input and output features to split the tensor into a number of sub-tensors, each of which is then separately approximated;5) fine-tuning of approximate models to (partially) recover the lost accuracy.Also, the authors consider two approximation criteria (Sect. 3.1):1) diagonal Mahalanobis metric, defined by the importance of the individual weights, computed by backprop;2) data covariance metric, which takes into consider the distribution of the layer inputs.The proposed speed-up methods are applied to the first and second convolutional layers of the state-of-the-art ConvNet architecture [8], and they are shown to speed-up the computation of these layers (but not the whole network) by a factor of 2-3, with a little loss of accuracy. SVD decomposition is also applied to the fully-connected layers of the network to reduce the number of parameters.== Positive points ==1. The problem of speeding up the computation of large ConvNets is relevant.2. The proposed methods for speeding-up ConvNets are novel.== Negative points ==The main problem of the paper is that even though a multitude of methods is proposed, their evaluation is superficial, which makes it hard to understand how these methods compare to each other. Also, the effect of methods' parameters is not properly explored. It would be really helpful to have a "take home" message for deep learning practitioners, which provides clear recipes on how to use the proposed methods. For that purpose, the authors might want to add an "accuracy vs speed-up" plot for the whole net (not just two speeded-up layers), with all approximations combined. Also, the description of some of the methods is too concise. A detailed list of issues and questions is given below.== Detailed comments ==1. Sect. 3.1: the proposed distance metrics are not compared to each other (and the Frobenius norm baseline) in the experiments. Fig. 4 only contains the results of the data covariance distance metric, the Mahalanobis metric (and the Frobenius norm) results do not seem to be reported. By the way, what is denoted as "Original" in Fig. 4?2. l.128, (1): which value of h did you use? Why not to compute the derivatives of the correct class score w.r.t. the approximated weights?3. l.144-153: the description is too concise, it should be extended (especially if this is the method used in the experiments).4. l.236, l.201 mention that the biclustering approximation is used for higher convolution layers. However, it's only applied to the first two layers. If the approximation was evaluated on higher levels, these results should be reported; otherwise, these statements should be corrected.5. l.236-242: is it true that each sub-tensor corresponds to a subset of input-output channels, and the subsets do not overlap, i.e. each channel is "covered" by a single sub-tensor? The authors should elaborate on how the result is computed by including a formal equation, instead of just referring to Fig. 1(b).6. l.241-242 ("each sub-tensor contains similar elements"): do you mean similar in the sense of the distances (2) or (3)? Are there any theoretical reasons why each cluster lives on a low-rank manifold, and is thus easier to approximate than the original tensor?7. Sect. 4.1.2: two approximations of Sect. 3.2.2 are not compared to each other without biclustering. It would be helpful to see the benefits of biclustering.8. l.362-365 ("we explored various configurations"): only the results corresponding to 2 combinations of G and H are given in the paper and the supp. material, which is not sufficient. In Fig. 4, why are different values of G, H, K, K1, and K2 used for the CPU and GPU implementations? How are they chosen?9. l.376 ("performance is restored"): there is no guarantee that the performance can be fully restored by fine-tuning. It would be helpful to see the results with and without fine-tuning.10. Sect. 4.3: how does SVD on the fully-connected layers affect the evaluation speed?11. l.269 ("4 convolutional layers"): isn't it supposed to be 5, if the configuration of [8] is used?12. l.285 ("20K validation images"): why not to report the performance on all 50K validation images, as typically used in the literature?13. Speed-up of the whole net: it would be helpful to see the speed-up of the whole net, rather than just the first two layers. According to Table 1 of the Supp. material, it seems that x2 speed-up of the first two conv. layers will result in only 22% reduction of the evaluation time of the whole net on the GPU?14. Relation to [6]: it is understood that this work was developed in parallel to [6] (recently published at BMVC-2014). While the comparison with [6] is not strictly required, it would be interesting to see it in the final version, if this paper gets accepted. Interestingly, [6] report 2-4 times speed-up for the *whole* network, but obviously the ConvNet configurations considered in this submission and [6] are different, so a direct comparison is not possible at the moment, and the authors are encouraged to add one.== Typos ==l. 214: "consider consider"l. 211, l.315 "Figure 4.1.1." -> "Figure 2"