Learning Curves for Deep Neural Networks: A field theory perspective
A series of recent works established a rigorous correspondence between very wide deep neural networks (DNNs), trained in a particular manner, and noiseless Bayesian Inference with a certain Gaussian Process (GP) known as the Neural Tangent Kernel (NTK). Here we extend a known field-theory formalism for GP inference to get a detailed understanding of learning-curves in DNNs trained in the regime of this correspondence (NTK regime). In particular, a renormalization-group approach is used to show that noiseless GP inference using NTK, which lacks a good analytical handle, can be well approximated by noisy GP inference on a related kernel we call the renormalized NTK. Following this, a perturbation-theory analysis is carried in one over the dataset-size yielding analytical expressions for the (fixed-teacher/fixed-target) leading and sub-leading asymptotics of the learning curves. At least for uniform datasets, a coherent picture emerges wherein fully-connected DNNs have a strong implicit bias towards functions which are low order polynomials of the input.
This theoretical paper exploits a recent rigorous correspondence between very wide DNNs (trained in a certain way) and Neural Tangent Kernel (a case of Gaussian Process-based noiseless Bayesian Inference). A field-theory formalism was developed for Gaussian Processes (2001). Here it is thus extended to the NTK case. There are 3 important theoretical results which are both proven and backed by numerical confirmation. These results, in particular the first, provide a very accurate prediction for the learning curve of some models. The paper is well situated within this literature. I am not very knowledgeable about NTK or even Gps, however I understand the challenges of understanding DNNs and I am familiar with field theory and renormalization group. Given the importance and quality of the results, and the overall quality and clarity of this (dense) paper, I recommend acceptation without hestiation. There are a couple of points however that could be improved, that would make the paper more useful for the community. Given the density of results in the paper, I would relax the length constraint, allowing up to 9 or 10 pages if possible, to add more explanations (not computations). I would like the paper to present more explicitly how the regression target labels g(x) are generated. Maybe it is said but I couldn’t easily understand, for sure, how they are generated. Also, please explain early enough what is meant by uniform dataset (I understood it simply means the data x is drawn uniformly at random over a manifold, here this manifold is often the d-dimensional hypersphere). Claim II states that ‘...lead to clear relations between deep fully-connected networks and polynomial regression’’. This is, I believe, supported by theoretical proof and numerical double-check, however it is not discussed enough for the abstract’s promise to be fulflled ‘a coherent picture emerges wherein fully-connected DNNs ...’. I think this point deserves a more ample discussion in section 7. More generally, the claims in the introduction or at the end of section 3 are stated rather explicitly, but very densely, and the careful reader can get the hypothesis of each result from the text. However for the sake of ease of read of less patient readers, I think it would be appropriate to have, somewhere, a more self-contained description of the results’ list. This paper is technical and some readers will be interested of simply knowing the hypothesis made and type of results obtained. For instance, the sentence ‘‘They [results] hold without any limitations on the dataset or the kernel and yield a variant of the EK result along with its sub-leading correction.’’ is misleading: as stated in the previous sentence in the text, this is for the fixed-teacher learning curve, etc. Please try to explain a bit more the intuition behind renormalization / trimming terms q>r (r integer fixed, the higher the less approximated). More specifically, it is not very clear to me how it can be interpreted in terms of how we look at the data. You mention (x.x’)^r being negligible or not depending on r,d etc, but I wounder if there is some kind of simple ‘geometrical’ interpretation (is it a coarse graining of the data in angular space, the ‘high energy’ eigenvalues corresponding to the high frequency, high resolution distinction between very close angles ?). On that point I am a bit lost and it’s a pity because your results are strong and rely on few, rather simple/elegant assumptions (which call for some intuitive understanding). Could you explain intuitively, to the inexperienced reader, why the noiseless case is harder to deal with than the finite-noise one ? In appendix D, it is mentioned that you need averaging over about 10 samples to have a decent average. For a single realization of a N-sized training set, there is additional variation (Adding or subtracting to the error epsilon). Given actual experiments are typically performed for a single realization of the data, I think this point should be mentioned in the main text more explicitly. Ideally, you could add error bars to the data, accounting for the dispersion inherent to a single-realization case. In early section 3, a short definition of a GP should be provided (there, or before). Around Eq. 2, you should specify the interpretation of P_0[f] After Eq. 3, ‘where F is some functional of f.’ I would add: ‘[where F is some functional of f,] for instance Eq. 2.’ The derivation of eq 6 is not obvious. You do detail it in the appendix, but forgot to cite the appendix ! ‘Notably none of these cited results apply in any straightforward manner in the NTK-regime.’ : could you quickly explain why (no matched priors ? Noise ?) ‘‘The d^-l scaling of eigenvalues’’ : at this point, the variable ‘l’ had not been defined. ‘‘notably cross-talk between features has been eliminated’’ : has it been eliminated or does it simply become constant ? ‘3% accuracy’ [relating to figure 1] I understand the idea but accuracy seems misleading. I would replace everywhere with something like ‘relative mismatch’. OR explain better why you call this accuracy: usually a high accuracy is preferred, and here you are proud with this very low imprecision of 3% ‘’Taking the leading order term one obtains the aforementioned EK result with N’’ : maybe (just a suggestion here) you could recall it here, given it was in page 1 (and in-line). Appendix B: could you explain why this difference increases with N ? I would have expected this kind of quantity to decrease with N. Appendix F: there are typos in the r.h.s. in the first line. sum_j f_j phi_j (I think). Appendix G: ‘noisy targets’ : you mean fully random or Kernel + some degree of noise with variance sigma^2 ? I think it’s the first time you use this phrasing. Appendix G: you denote partial / partial alpha for the functional derivative. I would replace with delta to stress out it is a functional and not regular derivative. Beyond appendix G.1 : I confess I didn’t have time to read it. Despite the overall quality of the text, there are a number of wrong singular/plural matchings, which can easily be corrected. Here are some of them, with other typos as well: ‘Furthermore since our aim was to predict what the DNNs would predict rather [THAN?] reach SOTA predictions’ ‘a factor of a factor of about 3.’ as do for – > as we do for uniformally - > uniformly
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Learning Curves for Deep Neural Networks: A field theory perspective
A series of recent works established a rigorous correspondence between very wide deep neural networks (DNNs), trained in a particular manner, and noiseless Bayesian Inference with a certain Gaussian Process (GP) known as the Neural Tangent Kernel (NTK). Here we extend a known field-theory formalism for GP inference to get a detailed understanding of learning-curves in DNNs trained in the regime of this correspondence (NTK regime). In particular, a renormalization-group approach is used to show that noiseless GP inference using NTK, which lacks a good analytical handle, can be well approximated by noisy GP inference on a related kernel we call the renormalized NTK. Following this, a perturbation-theory analysis is carried in one over the dataset-size yielding analytical expressions for the (fixed-teacher/fixed-target) leading and sub-leading asymptotics of the learning curves. At least for uniform datasets, a coherent picture emerges wherein fully-connected DNNs have a strong implicit bias towards functions which are low order polynomials of the input.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This theoretical paper exploits a recent rigorous correspondence between very wide DNNs (trained in a certain way) and Neural Tangent Kernel (a case of Gaussian Process-based noiseless Bayesian Inference). A field-theory formalism was developed for Gaussian Processes (2001). Here it is thus extended to the NTK case. There are 3 important theoretical results which are both proven and backed by numerical confirmation. These results, in particular the first, provide a very accurate prediction for the learning curve of some models. The paper is well situated within this literature. I am not very knowledgeable about NTK or even Gps, however I understand the challenges of understanding DNNs and I am familiar with field theory and renormalization group. Given the importance and quality of the results, and the overall quality and clarity of this (dense) paper, I recommend acceptation without hestiation. There are a couple of points however that could be improved, that would make the paper more useful for the community. Given the density of results in the paper, I would relax the length constraint, allowing up to 9 or 10 pages if possible, to add more explanations (not computations). I would like the paper to present more explicitly how the regression target labels g(x) are generated. Maybe it is said but I couldn’t easily understand, for sure, how they are generated. Also, please explain early enough what is meant by uniform dataset (I understood it simply means the data x is drawn uniformly at random over a manifold, here this manifold is often the d-dimensional hypersphere). Claim II states that ‘...lead to clear relations between deep fully-connected networks and polynomial regression’’. This is, I believe, supported by theoretical proof and numerical double-check, however it is not discussed enough for the abstract’s promise to be fulflled ‘a coherent picture emerges wherein fully-connected DNNs ...’. I think this point deserves a more ample discussion in section 7. More generally, the claims in the introduction or at the end of section 3 are stated rather explicitly, but very densely, and the careful reader can get the hypothesis of each result from the text. However for the sake of ease of read of less patient readers, I think it would be appropriate to have, somewhere, a more self-contained description of the results’ list. This paper is technical and some readers will be interested of simply knowing the hypothesis made and type of results obtained. For instance, the sentence ‘‘They [results] hold without any limitations on the dataset or the kernel and yield a variant of the EK result along with its sub-leading correction.’’ is misleading: as stated in the previous sentence in the text, this is for the fixed-teacher learning curve, etc. Please try to explain a bit more the intuition behind renormalization / trimming terms q>r (r integer fixed, the higher the less approximated). More specifically, it is not very clear to me how it can be interpreted in terms of how we look at the data. You mention (x.x’)^r being negligible or not depending on r,d etc, but I wounder if there is some kind of simple ‘geometrical’ interpretation (is it a coarse graining of the data in angular space, the ‘high energy’ eigenvalues corresponding to the high frequency, high resolution distinction between very close angles ?). On that point I am a bit lost and it’s a pity because your results are strong and rely on few, rather simple/elegant assumptions (which call for some intuitive understanding). Could you explain intuitively, to the inexperienced reader, why the noiseless case is harder to deal with than the finite-noise one ? In appendix D, it is mentioned that you need averaging over about 10 samples to have a decent average. For a single realization of a N-sized training set, there is additional variation (Adding or subtracting to the error epsilon). Given actual experiments are typically performed for a single realization of the data, I think this point should be mentioned in the main text more explicitly. Ideally, you could add error bars to the data, accounting for the dispersion inherent to a single-realization case. In early section 3, a short definition of a GP should be provided (there, or before). Around Eq. 2, you should specify the interpretation of P_0[f] After Eq. 3, ‘where F is some functional of f.’ I would add: ‘[where F is some functional of f,] for instance Eq. 2.’ The derivation of eq 6 is not obvious. You do detail it in the appendix, but forgot to cite the appendix ! ‘Notably none of these cited results apply in any straightforward manner in the NTK-regime.’ : could you quickly explain why (no matched priors ? Noise ?) ‘‘The d^-l scaling of eigenvalues’’ : at this point, the variable ‘l’ had not been defined. ‘‘notably cross-talk between features has been eliminated’’ : has it been eliminated or does it simply become constant ? ‘3% accuracy’ [relating to figure 1] I understand the idea but accuracy seems misleading. I would replace everywhere with something like ‘relative mismatch’. OR explain better why you call this accuracy: usually a high accuracy is preferred, and here you are proud with this very low imprecision of 3% ‘’Taking the leading order term one obtains the aforementioned EK result with N’’ : maybe (just a suggestion here) you could recall it here, given it was in page 1 (and in-line). Appendix B: could you explain why this difference increases with N ? I would have expected this kind of quantity to decrease with N. Appendix F: there are typos in the r.h.s. in the first line. sum_j f_j phi_j (I think). Appendix G: ‘noisy targets’ : you mean fully random or Kernel + some degree of noise with variance sigma^2 ? I think it’s the first time you use this phrasing. Appendix G: you denote partial / partial alpha for the functional derivative. I would replace with delta to stress out it is a functional and not regular derivative. Beyond appendix G.1 : I confess I didn’t have time to read it. Despite the overall quality of the text, there are a number of wrong singular/plural matchings, which can easily be corrected. Here are some of them, with other typos as well: ‘Furthermore since our aim was to predict what the DNNs would predict rather [THAN?] reach SOTA predictions’ ‘a factor of a factor of about 3.’ as do for – > as we do for uniformally - > uniformly
assistant
This theoretical paper exploits a recent rigorous correspondence between very wide DNNs (trained in a certain way) and Neural Tangent Kernel (a case of Gaussian Process-based noiseless Bayesian Inference). A field-theory formalism was developed for Gaussian Processes (2001). Here it is thus extended to the NTK case. There are 3 important theoretical results which are both proven and backed by numerical confirmation. These results, in particular the first, provide a very accurate prediction for the learning curve of some models. The paper is well situated within this literature. I am not very knowledgeable about NTK or even Gps, however I understand the challenges of understanding DNNs and I am familiar with field theory and renormalization group. Given the importance and quality of the results, and the overall quality and clarity of this (dense) paper, I recommend acceptation without hestiation. There are a couple of points however that could be improved, that would make the paper more useful for the community. Given the density of results in the paper, I would relax the length constraint, allowing up to 9 or 10 pages if possible, to add more explanations (not computations). I would like the paper to present more explicitly how the regression target labels g(x) are generated. Maybe it is said but I couldn’t easily understand, for sure, how they are generated. Also, please explain early enough what is meant by uniform dataset (I understood it simply means the data x is drawn uniformly at random over a manifold, here this manifold is often the d-dimensional hypersphere). Claim II states that ‘...lead to clear relations between deep fully-connected networks and polynomial regression’’. This is, I believe, supported by theoretical proof and numerical double-check, however it is not discussed enough for the abstract’s promise to be fulflled ‘a coherent picture emerges wherein fully-connected DNNs ...’. I think this point deserves a more ample discussion in section 7. More generally, the claims in the introduction or at the end of section 3 are stated rather explicitly, but very densely, and the careful reader can get the hypothesis of each result from the text. However for the sake of ease of read of less patient readers, I think it would be appropriate to have, somewhere, a more self-contained description of the results’ list. This paper is technical and some readers will be interested of simply knowing the hypothesis made and type of results obtained. For instance, the sentence ‘‘They [results] hold without any limitations on the dataset or the kernel and yield a variant of the EK result along with its sub-leading correction.’’ is misleading: as stated in the previous sentence in the text, this is for the fixed-teacher learning curve, etc. Please try to explain a bit more the intuition behind renormalization / trimming terms q>r (r integer fixed, the higher the less approximated). More specifically, it is not very clear to me how it can be interpreted in terms of how we look at the data. You mention (x.x’)^r being negligible or not depending on r,d etc, but I wounder if there is some kind of simple ‘geometrical’ interpretation (is it a coarse graining of the data in angular space, the ‘high energy’ eigenvalues corresponding to the high frequency, high resolution distinction between very close angles ? ). On that point I am a bit lost and it’s a pity because your results are strong and rely on few, rather simple/elegant assumptions (which call for some intuitive understanding). Could you explain intuitively, to the inexperienced reader, why the noiseless case is harder to deal with than the finite-noise one ? In appendix D, it is mentioned that you need averaging over about 10 samples to have a decent average. For a single realization of a N-sized training set, there is additional variation (Adding or subtracting to the error epsilon). Given actual experiments are typically performed for a single realization of the data, I think this point should be mentioned in the main text more explicitly. Ideally, you could add error bars to the data, accounting for the dispersion inherent to a single-realization case. In early section 3, a short definition of a GP should be provided (there, or before). Around Eq. 2, you should specify the interpretation of P_0[f] After Eq. 3, ‘where F is some functional of f.’ I would add: ‘[where F is some functional of f,] for instance Eq. 2.’ The derivation of eq 6 is not obvious. You do detail it in the appendix, but forgot to cite the appendix ! ‘Notably none of these cited results apply in any straightforward manner in the NTK-regime.’ : could you quickly explain why (no matched priors ? Noise ?) ‘‘The d^-l scaling of eigenvalues’’ : at this point, the variable ‘l’ had not been defined. ‘‘notably cross-talk between features has been eliminated’’ : has it been eliminated or does it simply become constant ? ‘3% accuracy’ [relating to figure 1] I understand the idea but accuracy seems misleading. I would replace everywhere with something like ‘relative mismatch’. OR explain better why you call this accuracy: usually a high accuracy is preferred, and here you are proud with this very low imprecision of 3% ‘’Taking the leading order term one obtains the aforementioned EK result with N’’ : maybe (just a suggestion here) you could recall it here, given it was in page 1 (and in-line). Appendix B: could you explain why this difference increases with N ? I would have expected this kind of quantity to decrease with N. Appendix F: there are typos in the r.h.s. in the first line. sum_j f_j phi_j (I think). Appendix G: ‘noisy targets’ : you mean fully random or Kernel + some degree of noise with variance sigma^2 ? I think it’s the first time you use this phrasing. Appendix G: you denote partial / partial alpha for the functional derivative. I would replace with delta to stress out it is a functional and not regular derivative. Beyond appendix G.1 : I confess I didn’t have time to read it. Despite the overall quality of the text, there are a number of wrong singular/plural matchings, which can easily be corrected. Here are some of them, with other typos as well: ‘Furthermore since our aim was to predict what the DNNs would predict rather [THAN?] reach SOTA predictions’ ‘a factor of a factor of about 3.’ as do for – > as we do for uniformally - > uniformly