Recent research shows that for training with 2 loss, convolutional neural networks (CNNs) whose width (number of channels in convolutional layers) goes to infinity correspond to regression with respect to the CNN Gaussian Process kernel (CNN-GP) (Novak et al., 2019) if only the last layer is trained, and correspond to regression with respect to the Convolutional Neural Tangent Kernel (CNTK) if all layers are trained. An exact algorithm to compute CNTK (Arora et al., 2019) yielded the finding that classification accuracy of CNTK on CIFAR-10 is within 6-7% of that of the corresponding CNN architecture (best figure being around 78%) which is interesting performance for a fixed kernel. Here we show how to significantly enhance the performance of these kernels using two ideas. (1) Modifying the kernel using a new operation called Local Average Pooling (LAP) which preserves efficient computability of the kernel and inherits the spirit of standard data augmentation using pixel shifts. Earlier papers were unable to incorporate naive data augmentation because of the quadratic training cost of kernel regression. This idea is inspired by Global Average Pooling (GAP), which we show for CNN-GP and CNTK is equivalent to full translation data augmentation. (2) Representing the input image using a pre-processing technique proposed by Coates et al. (2011), which uses a single convolutional layer composed of random image patches. On CIFAR-10, the resulting kernel, CNN-GP with LAP and horizontal flip data augmentation, achieves 89% accuracy, matching the performance of AlexNet (Krizhevsky et al., 2012) , and outperforms the best previous classifier that is not a trained neural network (Mairal, 2016). Similar improvements are obtained for Fashion-MNIST.
This paper shows that there is a one-to-one correspondence between pixel-shift based data augmentation and average pooling operations in CNN-NNGP/NTK based ridge regression. Interestingly, the authors show that standard average pooling + flatten can lead to a better performance than simple global average pooling. This paper further shows that using the data pre-processing step proposed in (Coates et al., 2011) can boost performance of CNN-NNGP/NTK based ridge regression by ~7% which allowed the authors to achieve classification accuracy in high 80s which is AFAIK SOTA on CIFAR-10 when not using learned representations.
My current assessment of the paper is “weak accept”. There are two main reasons why I am on the verge of recommending rejection of this paper: (1) I believe that the experiment evaluation is not done entirely correctly leading to inflation of the reported results (my guesstimate is by ~0.5-2%)---please see my “Major comments”. If this is not fixed, I am very likely to downgrade my score. (2) While the observation of the relationship between pixel-shifts and average pooling is very nice (which is why my current score is “weak accept”), it seems that most of the improvement comes from application of the pre-processing step of Coates et al. (2011) (seems like a ~7% improvement!). Given the large computational cost of CNN-NNGP/NTK (authors say about 1000 GPU hours), I wonder whether a simpler algorithm like some of the newer variants of boosting combined with the Coates et al. algorithm wouldn’t also perform at around 87-88% like CNN-NNGP/NTK (given the baseline 85-86% accuracy of the Coates et al. (2011) algorithm reported by the authors).
Major comments:
- Can you please clarify why you decided to give a new name (Box Blur) to standard average pooling? Why not just use the existing name?
- I believe that the way you report results in all the tables (i.e., tables 1-6) and the text based upon them is flawed. The right approach would be to select the hyperparameters “c” and “d” on a validation set, and then report the performance with these hyperparameters on the test set. While the experiments are somewhat rescued by the fact that you report results for (almost) all the possible hyperparameter settings (which allows us to see samples from the population distribution of the generalisation error), type-setting the best results in boldface and thus implying that these are valid estimates of the generalisation error is not appropriate since you are effectively selecting the best hyper-parameters on the test set! Unfortunately, I cannot accept these results to be published “as-is”. While re-running the experiments with hyperparameter selection on validation set is already a somewhat imperfect solution, I am not sure I can see a better way forward. However, I do understand that this could be prohibitively expensive in which case I would like to ask you to suggest an alternative solution please (of course, other reviewers are welcome to chime in as well)?!
- While most of the paper is about Local Average Pooling (LAP) and the equivalence between averaging and pixel shifts, the experimental results seem to show that most of the improvement comes from the use of Coates et al.’s preprocessing step. Could you please run the experiments in tables 3 and 4 with c=0 and c=32 to see what the effect of the preprocessing is without LAP?
- In sect.6.3, you say “Our experiment illustrates that even with a fixed last FC layer, using GAP could improve the performance of CNN, and challenges the conjecture that GAP reduces the number of parameters in the last fully-connected layer and thus avoids overfitting.” I am not sure I see why fixing the last FC layer should provide more convincing evidence than training it? I do not know the conjecture to which you refer but from your description, the overfitting without GAP should occur because the FC layer has more parameters than with GAP?! If this is true, then the overfitting would happen in the last layer (due to the large number of parameters) which you have (at least partially) prevented by not training it?! Can you clarify and also report the results of this experiment with all the layers trained please?
- In Appendix D, you say that you have used lambda = 10^{-5} for all configurations. How have you selected this particular value please? Do you have a sense of how far from optimal this value is for all the different configurations (or at least for NTK vs NNGP models---in my experience, the optimal setting between the two can differ quite a bit)?
Minor comments:
- In the abstract and throughout the paper, you claim that the cost of kernel regression is quadratic. AFAIK without any approximations, the cost is cubic (or O(n^{2.67}) to be more precise). Please clarify.
- In par.1 on p.1, you say “convolutional neural networks (CNNs) whose width (number of channels in convolutional layers) **goes to infinity**” (emphasis mine) and cite the Jacot et al. (2018) paper. AFAIK this paper only works with infinite networks but does not actually prove that **deep** networks of finite width (in each layer) converge to the NTK limit; IMHO you should cite the Allen-Zhu et al. (2018) and Du et al. (2018) papers from your references for that result. Based on p.2 (end of par.2 in sect.2), you seem to be aware of this distinction but cite Arora et al. (2019) instead of these two; I would suggest either citing Allen-Zhu et al. and Du et al. only, or citing all three as the Arora et al. paper came out later than the first versions of the other two paper which AFAIK already contained all the necessary derivations (even if the words “Neural Tangent Kernel” were not spelled out there).
- Also in par.1 on p.1, you say that Arora et al. (2019) was the first to provide an algorithm to compute the CNTK kernel which is a bit of a stretch given that both Garriga-Alonso et al. (2019) and Novak et al. (2019) have implemented the CNTK kernel in their experiments. AFAIK the claim in (Arora et al., 2019) is that they provided first **efficient** implementation of the CNTK-GAP kernel which should be made clearer in the next revision of your paper.
- On p.2, you say “These kernels correspond to neural networks where only the last layer is trained.” In reality, the correspondence is not exact for finite networks because the induced kernel will not be exactly equal to the one at the limit.
- Bottom of p.2, “Global Average Pooling (GAP) is proposed” -> “... was proposed”.
- Top of p.3, “..., and GAP is more robust” -> “..., and that GAP is more robust”.
- On p.3 in the “Padding Schemes” paragraph, do you mean to assume that the input image has only a single channel (not necessary later)?
- On p.4, I am slightly confused by your definition of the “augmented kernel”. Specifically, it does not seem K^G (x , x’) = K^G (x’, x) holds in general. Can you please clarify? If there’s no symmetry, I do not think it necessary to use a different name, but perhaps a clarifying note would be beneficial to the reader?!
- On p.5, fig.1 is too small when printed and one needs to use the computer screen to see what is depicted; given the amount of white space around, can you please try to make the images larger (you can perhaps also only include 2 or 4 images instead of 16 which will give you additional space)?
- On p.5, you say that for small “c”, circular padding will not create unrealistic images. Looking at fig.1b, it seems like the images are not as unrealistic as in fig.1a but human eye can still tell they are not realistic (potentially even more so with other images than the one selected for this figure). I am not sure whether there is a reason to assume this issue does not affect CNNs too?! Further, I am not convinced the motivation is correct in the first place given that the optimal “c” for CIFAR-10 is 12 which will presumably create clearly unrealistic images; perhaps it would be best to omit this motivation?!
- On p.6, you claim “Another advantage of LAP is that it **does not incur any extra computational cost**” (emphasis mine) while at the next line you say that there is a constant additional computational cost. Perhaps say that the extra computational cost is relatively small?
- It might be nice to swap tables 3 and 4 so at least the results for NNGP are next to each other. Even better would be the current table 3 was closer to table 1 to achieve the same effect for NTK.
- I am not sure I fully understand the description in sect.6.3: isn’t the number of channels on the input irrelevant after computation of the kernel in the first layer? In other words, why have you opted to use only 2,048 patches in your experiments and not 32,000 or 256,000 as used by Recht et al. (2019)? Do you have an estimate of how different could the performance of NNGP/NTK be with the larger number of features? Do you know what is the performance of Coates et al.’s algorithm with only 2,048 features? Relatedly, do you know how AlexNet would perform if its PCA data augmentation was replaced by the Coates et al.’s feature extractor?