Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm
How many training data are needed to learn a supervised task? It is often observed that the generalization error decreases as where is the number of training examples and an exponent that depends on both data and algorithm. In this work we measure when applying kernel methods to real datasets. For MNIST we find and for CIFAR10 . Remarkably, is the same for regression and classification tasks, and for Gaussian or Laplace kernels. To rationalize the existence of non-trivial exponents that can be independent of the specific kernel used, we introduce the Teacher-Student framework for kernels. In this scheme, a Teacher generates data according to a Gaussian random field, and a Student learns them via kernel regression. With a simplifying assumption --- namely that the data are sampled from a regular lattice --- we derive analytically for translation invariant kernels, using previous results from the kriging literature. Provided that the Student is not too sensitive to high frequencies, depends only on the training data and their dimension. We confirm numerically that these predictions hold when the training points are sampled at random on a hypersphere. Overall, our results quantify how smooth Gaussian data should be to avoid the curse of dimensionality, and indicate that for kernel learning the relevant dimension of the data should be defined in terms of how the distance between nearest data points depends on . With this definition one obtains reasonable effective smoothness estimates for MNIST and CIFAR10.
In order to rationalize the existence of non-trivial exponents that can be independent of the specific kernel used, this paper introduces the Teacher-Student framework for kernels. In this scheme, a Teacher generates data according to a Gaussian random field, and a Student learns them via kernel regression. Theresults quantify how smooth Gaussian data should be to avoid the curse of dimensionality, and indicate that for kernel learning the relevant dimension of the data should be defined in terms of how the distance between nearest data points depends on sample numbers. The paper is well written, tghe major issue of this paper is the lack of comparison with other previous methods. Therefore, the efficacy of the proposed model can not be well demontrated.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm
How many training data are needed to learn a supervised task? It is often observed that the generalization error decreases as where is the number of training examples and an exponent that depends on both data and algorithm. In this work we measure when applying kernel methods to real datasets. For MNIST we find and for CIFAR10 . Remarkably, is the same for regression and classification tasks, and for Gaussian or Laplace kernels. To rationalize the existence of non-trivial exponents that can be independent of the specific kernel used, we introduce the Teacher-Student framework for kernels. In this scheme, a Teacher generates data according to a Gaussian random field, and a Student learns them via kernel regression. With a simplifying assumption --- namely that the data are sampled from a regular lattice --- we derive analytically for translation invariant kernels, using previous results from the kriging literature. Provided that the Student is not too sensitive to high frequencies, depends only on the training data and their dimension. We confirm numerically that these predictions hold when the training points are sampled at random on a hypersphere. Overall, our results quantify how smooth Gaussian data should be to avoid the curse of dimensionality, and indicate that for kernel learning the relevant dimension of the data should be defined in terms of how the distance between nearest data points depends on . With this definition one obtains reasonable effective smoothness estimates for MNIST and CIFAR10.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



In order to rationalize the existence of non-trivial exponents that can be independent of the specific kernel used, this paper introduces the Teacher-Student framework for kernels. In this scheme, a Teacher generates data according to a Gaussian random field, and a Student learns them via kernel regression. Theresults quantify how smooth Gaussian data should be to avoid the curse of dimensionality, and indicate that for kernel learning the relevant dimension of the data should be defined in terms of how the distance between nearest data points depends on sample numbers. The paper is well written, tghe major issue of this paper is the lack of comparison with other previous methods. Therefore, the efficacy of the proposed model can not be well demontrated.
assistant
In order to rationalize the existence of non-trivial exponents that can be independent of the specific kernel used, this paper introduces the Teacher-Student framework for kernels. In this scheme, a Teacher generates data according to a Gaussian random field, and a Student learns them via kernel regression. Theresults quantify how smooth Gaussian data should be to avoid the curse of dimensionality, and indicate that for kernel learning the relevant dimension of the data should be defined in terms of how the distance between nearest data points depends on sample numbers. The paper is well written, tghe major issue of this paper is the lack of comparison with other previous methods. Therefore, the efficacy of the proposed model can not be well demontrated.