A series of recent works established a rigorous correspondence between very wide deep neural networks (DNNs), trained in a particular manner, and noiseless Bayesian Inference with a certain Gaussian Process (GP) known as the Neural Tangent Kernel (NTK). Here we leverage this correspondence to provide explicit analytical predictions for the learning curves of DNNs on any fixed target function. Focusing on datasets whose input measure is uniform on a hypersphere, our predictions match experimental curves very well and reveal a strong implicit bias towards functions which are low order polynomials of the input. Two novel analytical tools underlay our approach. First a renormalization-group approach is used to show that noiseless GP inference using NTK, which lacks a good analytical handle, can be well approximated by noisy GP inference on a related kernel we call the renormalized NTK. Second a field theory reformulation of the problem allows a controlled perturbative expansion of the test error in the inverse dataset size. We believe our approach lays the foundations for a more precise quantitative understanding of DNNs.
This theoretical paper exploits a recent rigorous correspondence between very wide DNNs (trained in a certain way) and Neural Tangent Kernel (a case of Gaussian Process-based noiseless Bayesian Inference).  
A field-theory formalism was developed for Gaussian Processes (2001). Here it is thus extended to the NTK case. There are 3 important theoretical results which are both proven and backed by numerical confirmation.  These results, in particular the first, provide a very accurate prediction for the learning curve of some models.  The paper is well situated within this literature.  I am not very knowledgeable about NTK or even Gps, however I understand the challenges of understanding DNNs and I am familiar with field theory and renormalization group.
Given the importance and quality of the results, and the overall quality and clarity of this (dense) paper, I recommend acceptation without hestiation.
There are a couple of points however that could be improved, that would make the paper more useful for the community.
Given the density of results in the paper, I would relax the length constraint, allowing up to 9 or 10 pages if possible, to add more explanations (not computations).
I would like the paper to present more explicitly how the regression target labels g(x) are generated. Maybe it is said but I couldn’t easily understand, for sure, how they are generated.
Also, please explain early enough what is meant by uniform dataset (I understood it simply means the data x is drawn uniformly at random over a manifold, here this manifold is often the d-dimensional hypersphere).
Claim II states that ‘...lead to clear relations between deep fully-connected networks and polynomial
regression’’. This is, I believe, supported by theoretical proof and numerical double-check, however it is not discussed enough for the abstract’s promise to be fulflled ‘a coherent picture emerges wherein fully-connected DNNs ...’.
I think this point deserves a more ample discussion in section 7.
More generally, the claims in the introduction or at the end of section 3 are stated rather explicitly, but very densely, and the careful reader can get the hypothesis of each result from the text. 
However for the sake of ease of read of less patient readers, I think it would be appropriate to have, somewhere, a more self-contained description of the results’ list. This paper is technical and some readers will be interested of simply knowing the hypothesis made and type of results obtained.
For instance, the sentence ‘‘They [results] hold without any limitations on the dataset or the kernel and yield a variant of the EK result along with its sub-leading correction.’’ is misleading: as stated in the previous sentence in the text, this is for the fixed-teacher learning curve, etc.
Please try to explain a bit more the intuition behind renormalization / trimming terms q>r (r integer fixed, the higher the less approximated).  More specifically, it is not very clear to me how it can be interpreted in terms of how we look at the data. You mention (x.x’)^r being negligible or not depending on r,d etc, but I wounder if there is some kind of simple ‘geometrical’ interpretation (is it a coarse graining of the data in angular space, the ‘high energy’ eigenvalues corresponding to the high frequency, high resolution distinction between very close angles ?).  On that point I am a bit lost and it’s a pity because your results are strong and rely on few, rather simple/elegant assumptions (which call for some intuitive understanding).
Could you explain intuitively, to the inexperienced reader, why the noiseless case is harder to deal with than the finite-noise one ?
In appendix D, it is mentioned that you need averaging over about 10 samples to have a decent average. For a single realization of a N-sized training set, there is additional variation (Adding or subtracting to the error epsilon).  Given actual experiments are typically performed for a single realization of the data, I think this point should be mentioned in the main text more explicitly. Ideally, you could add error bars to the data, accounting for the dispersion inherent to a single-realization case.
In early section 3, a short definition of a GP should be provided (there, or before).
Around Eq. 2, you should specify the interpretation of P_0[f]
After Eq. 3, ‘where F is some functional of f.’ I would add: ‘[where F is some functional of f,] for instance Eq. 2.’
The derivation of eq 6 is not obvious. You do detail it in the appendix, but forgot to cite the appendix !
‘Notably none of these cited results apply in any straightforward manner in the NTK-regime.’ : could you quickly explain why (no matched priors ? Noise ?) 
‘‘The d^-l scaling of eigenvalues’’ : at this point, the variable ‘l’ had not been defined.
‘‘notably cross-talk between features has been eliminated’’ : has it been eliminated or does it simply become constant ? 
‘3% accuracy’ [relating to figure 1]
I understand the idea but accuracy seems misleading. I would replace everywhere with something like ‘relative mismatch’.  OR explain better why you call this accuracy: usually a high accuracy is preferred, and here you are proud with this very low imprecision of 3%
‘’Taking the leading order term one obtains the aforementioned EK result with N’’ : maybe (just a suggestion here) you could recall it here, given it was in page 1 (and in-line).
Appendix B:  could you explain why this difference increases with N ? I would have expected this kind of quantity to decrease with N.
Appendix F: there are typos in the r.h.s. in the first line.
\sum_j f_j \phi_j  (I think).
Appendix G: ‘noisy targets’ : you mean fully random or Kernel + some degree of noise with variance sigma^2 ?  I think it’s the first time you use this phrasing.
Appendix G: you denote \partial / \partial \alpha for the functional derivative. I would replace with \delta to stress out it is a functional and not regular derivative.
Beyond	 appendix G.1 : I confess I didn’t have time to read it.
Despite the overall quality of the text, there are a number of wrong singular/plural matchings, which can easily be corrected. Here are some of them, with other typos as well: 
‘Furthermore since our aim was to predict what the DNNs would predict rather [THAN?] reach SOTA predictions’
‘a factor of a factor
of about 3.’
as do for – > as we do for
uniformally - > uniformly