How many training data are needed to learn a supervised task? It is often observed that the generalization error decreases as n −β where n is the number of training examples and β an exponent that depends on both data and algorithm. In this work we measure β when applying kernel methods to real datasets. For MNIST we find β ≈ 0.4 and for CIFAR10 β ≈ 0.1. Remarkably, β is the same for regression and classification tasks, and for Gaussian or Laplace kernels. To rationalize the existence of non-trivial exponents that can be independent of the specific kernel used, we introduce the Teacher-Student framework for kernels. In this scheme, a Teacher generates data according to a Gaussian random field, and a Student learns them via kernel regression. With a simplifying assumption -namely that the data are sampled from a regular lattice -we derive analytically β for translation invariant kernels, using previous results from the kriging literature. Provided that the Student is not too sensitive to high frequencies, β depends only on the training data and their dimension. We confirm numerically that these predictions hold when the training points are sampled at random on a hypersphere. Overall, our results quantify how smooth Gaussian data should be to avoid the curse of dimensionality, and indicate that for kernel learning the relevant dimension of the data should be defined in terms of how the distance between nearest data points depends on n. With this definition one obtains reasonable effective smoothness estimates for MNIST and CIFAR10.
This paper studies, empirically and theoretically, the learning rates of (shift-invariant) kernel learners in a misspecified setting. In the well-specified setting, the rate of kernel learners is at least $n^{-1/2}$, and in a misspecified setting assuming only Lipschitz targets, the rate is $n^{-1/d}$. Neither seems to match the experimental rate on MNIST and CIFAR-10; this paper proposes a theoretical model that can more-or-less match the experimental rate with essentially-reasonable assumptions.
My main complaint is on the basic setting of the work: in your motivation, you say "it is nowadays part of the lore that there exist kernels whose performance is nearly comparable to deep networks." The main such kernel, though, is the (convolutional) neural tangent kernel of Arora et al. (2019), which unlike the kernels you study here is not shift-invariant, and your theorems do not at all apply to this kernel. This is fine, but should probably be clearer in the description.
A related comment on your main theorem: your target function evaluated at every conceivable point (not just on a grid) is a sample from a Gaussian process. Samples from GPs with mean zero and covariance kernel $K_T$ almost surely are not in the RKHS $\mathcal H_T$, but they *are* almost surely in the RKHS of any kernel $K_R$ which nuclearly dominates $K_T$ (see Lukic and Beder, "Stochastic Processes with Sample Paths in Reproducing Kernel Hilbert Spaces", Trans. AMS 2001). If such a kernel exists, using it as the "student" kernel should give us a rate of at least $n^{-1/2}$ with standard results (with some slight details still to be worked out, but should be true). Thus, it seems that your theorem implies that for $\alpha_T < \frac32 d$, no such translation-invariant kernel $R$ exists. This might be already easy to see from a Fourier definition of nuclear dominance, I'm not sure, but if not it is something that seems of somewhat independent interest.
It is also notable that both your practical results and your theorem are for algorithms essentially without any regularization other than the choice of kernel: the regression setting is exact interpolation, and your soft-margin uses $C = 10^4$ so is "almost" a hard-margin SVM. This is also fine – interpolation methods have seen a lot of interest of late, and certainly can perform well. But it's not the typical setting, and it would be interesting to see if the curves of Figure 1 look different when using e.g. a cross-validated setting for the amount of regularization.
Another complaint: you argue that applying Theorem 1 with this particular notion of effective dimension seems to give good results, but at least as it's stated, Theorem 1 doesn't actually apply with effective dimension, only ambient dimension. Is it possible to prove Theorem 1 with an appropriate version of effective dimension? I didn't carefully check the proof, but from your outlined sketch it seems like it might be only a small change.
Empirically, your investigations are nice, but it would be good to consider some other shift-invariant kernels as well: inverse multiquadric, Matérn, or spline RBF kernels would be prominent options.
Overall: I think this is a worthwhile study with interesting results. The theoretical setting, though, is somewhat limited by its fundamental approach, and the experiments aren't as thorough as they could be. Also, honestly, I'm not sure ICLR is the best venue for it (if I had written this paper around this time, I probably would have submitted it to AISTATS; it's certainly not *off* topic for ICLR, but fairly distant from most work at it).
Some typos:
- Under (2): "man-square error."
- Under (25): "where where."