Principal Filter Analysis (PFA) is an easy to implement, yet effective method for neural network compression. PFA exploits the intrinsic correlation between filter responses within network layers to recommend a smaller network footprint. We propose two compression algorithms: the first allows a user to specify the proportion of the original spectral energy that should be preserved in each layer after compression, while the second is a heuristic that leads to a parameter-free approach that automatically selects the compression used at each layer. Both algorithms are evaluated against several architectures and datasets, and we show considerable compression rates without compromising accuracy, e.g., for VGG-16 on CIFAR-10, CIFAR-100 and ImageNet, PFA achieves a compression rate of 8x, 3x, and 1.4x with an accuracy gain of 0.4%, 1.4% points, and 2.4% respectively. In our tests we also demonstrate that networks compressed with PFA achieve an accuracy that is very close to the empirical upper bound for a given compression ratio. Finally, we show how PFA is an effective tool for simultaneous compression and domain adaptation.
This paper introduces an approach to compressing a trained neural network by looking at the correlation of the filter responses in each layer. Two strategies are proposed: one based on trying to preserve the energy of the original activations and one based on looking at the KL divergence between the normalized eigenvalues of the activation covariance matrix and the uniform distribution.
Strengths:
- The KL-divergence-based method is novel and has the advantage of not requiring to define any hyper-parameter.
- The results show the good behavior of the approach.
Weaknesses:
Method:
- One thing that bothers me is the spatial max pooling of the activations of convolutional layers. This means that is two filters have high responses on different regions of the input image, they will be treated as correlated. I do not understand the intuition behind this.
- In Section 2, the authors mention that other methods have also proposed to take the activation into account for pruning, but that they aim to minimize the reconstruction error of these activations. In fact, this is also what PFA-En does; for a given dimension, PCA gives the representation that minimizes the reconstruction error. Therefore, the connection between this method and previous works is stronger than claimed by the authors.
- While it is good that the KL-divergence-based method does not rely on any hyper-parameter, the function \psi used in Eq. 3 seems quite ad hoc. As such, there has also been some manual tuning of the method.
Experiments:
- In Table 1, there seems to be a confusion regarding how the results of FGA are reported. First, in (Peng et al., 2018), the %FLOPS is reported the other way around, i.e., the higher the better, whereas here the lower the better. Similarly, in (Peng et al., 2018), a negative \Delta in accuracy means an improved performance (as stated in the caption of their Table 2, where the numbers reported here were taken). As such, the numbers reported here, and directly taken from this work, are misinterpreted. 
- Furthermore, Peng et al., 2018 report much better compression results, with %FLOP compression going up to 88.58%. Why are these results not reported here? (To avoid any misunderstanding, I would like to mention that I am NOT an author of (Peng et al., 2018)).
- Many of the entries in Table 1 are empty due to the baselines not reporting results on these datasets or with the same network. This makes an actual comparison more difficult.
- Many compression methods report results on ImageNet. This would make this paper more convincing.
- While I appreciate the domain adaptation experiments, it would be nice to see a comparison with Masana et al., 2017, which also considers the problem of domain adaptation with network compression and, as mentioned in Section 2, also makes use of the activations to achieve compression.
Related work:
- It is not entirely clear to me why tensor factorization methods are considered being so different from the proposed approach. In essence, they also perform structured network pruning.
- The authors argue that performing compression after having trained the model is beneficial. This is in contrast with what was shown by Alvarez & Salzmann, NIPS 2017, where incorporating a low-rank prior during training led to higher compression rates.
- The authors list (Dai et al., 2018) as one of the methods that aim to minimize the reconstruction error of the activations. Dai et al., 2018 rely on the mutual information between the activations in different layers to perform compression. It is not entirely clear to me how this relates to reconstruction error.
Summary:
I do appreciate the idea of aiming for a hyper-parameter-free compression method. However, I feel that there are too many points to be corrected or clarified and too many missing experiments for this paper to be accepted to ICLR.
After Response:
I appreciate the authors' response, which clarified several of my concerns. I would rate this paper as borderline. My main concern now is that the current comparison with existing method still seems too incomplete, especially with ResNet architectures, to really draw conclusions. I would therefore encourage the authors to revise their paper and re-submit it to an upcoming venue.