An intriguing phenomenon observed during training neural networks is the spectral bias, where neural networks are biased towards learning less complex functions. The priority of learning functions with low complexity might be at the core of explaining generalization ability of neural network, and certain efforts have been made to provide theoretical explanation for spectral bias. However, there is still no satisfying theoretical result justifying the underlying mechanism of spectral bias. In this paper, we give a comprehensive and rigorous explanation for spectral bias and relate it with the neural tangent kernel function proposed in recent work. We prove that the training process of neural networks can be decomposed along different directions defined by the eigenfunctions of the neural tangent kernel, where each direction has its own convergence rate and the rate is determined by the corresponding eigenvalue. We then provide a case study when the input data is uniformly distributed over the unit sphere, and show that lower degree spherical harmonics are easier to be learned by over-parameterized neural networks.
This paper studies the training of overparametrized neural networks by gradient descent. More precisely, the authors consider the neural tangent regime (NTK regime). That is, the weights are chosen sufficiently large and the neural network is sufficiently overparametrized. It has been observed that in this scenario, the neural network behaves approximately like a linear function of its weights.
In this regime, the authors show that, the directions corresponding to larger eigenvalues of the neural tangent kernel are learned first. As this corresponds to learning lower-degree polynomials first, the authors claim that this explains the "spectral bias" observed in previous papers.
-I think that from a mathematical point of view, the main result of this paper is what one would expect intuitively: 
When performing gradient descent with quadratic loss where the function to be learnt is linear, it is common knowledge that convergence is faster on directions corresponding to larger singular values. Since in the NTK regime, the neural network can be approximated by a linear function around the initialization one expects the behavior predicted by the main results. From a theoretical perspective, I see the main contribution of the paper as making this statement precise.
-I am skeptical about some of the implications for practitioners, which are given by the authors: 
For example, on p.5 the authors write "Therefore, Theorem 3.2 theoretically explains the empirical observations given in Rahaman et al. (2018), and demonstrates that the difficulty of a function to be learned by neural network should be studied in the eigenspace of neural tangent kernel." To the best of my knowledge, it is unclear whether practitioners train neural networks in the NTK regime (see, e.g., [1]). Moreover, I am wondering whether some of the assumptions of their theorem are really met in practice. For example, the required sample size for higher order polynomials grows exponentially fast with the order and the required step size goes to zero exponentially fast. Does this really correspond to what is observed in practice? (Or is this a mere artifact of training in the NTK regime?)  Is this what one observes in the experiments by Ramahan?
I think the paper is not yet ready for being published.
 1. There are many typos. Here is an (very incomplete) list.
    -p. 2: "Su and Yang (2019)" improves the convergence..."
    -p. 2: "This theorem gives finer-grained control on error term's"
    -p. 2: "We present a more general results"
    -p. 4: "The variance follows the principal..."
    -p. 4: "...we will present Mercer decomposition in (the) next section."
2. I think that the presentation can be polished and many statements are somewhat unclear. For example, on p. 7 the authors write "the convergence rates [...] are exactly predicted by our theory in a qualitative sense."
    The meaning of this sentence is unclear to me. Does that mean in a quantitative sense? To be honest, only considering Fig. 1 I am not able to assess whether the convergence rates of the different components are truly linear. 
I decided for my rating of the paper because of the following reasons:
-I think that for a theory paper the results obtained by the authors are not enough, as they are rather direct consequences of the "near-linearity" of the neural network around the initialization.
-In my view, there is a huge gap between current theoretical results for deep learning and practice. For this reason, it is not problematic for me that it is unclear, what the results in this paper mean for practitioners. (Apart from that, results for the NTK regime are interesting in its own right.) However, in my view, one should explain the limitations of the theory more carefully.
-The presentation of the paper needs to be improved.
References:
[1] A note on lazy training in supervised differentiable programming. L Chizat, F Bach - arXiv preprint arXiv:1812.07956, 2018 
-----------------------------
I highly appreciate the authors' detailed response. However, I feel that the paper does not contain enough novelty to justify acceptance.
------
"Equation (8) in Arora et al., (2019b) only provides a bound on the whole residual vector, i.e., , and therefore cannot show different convergence rates along different directions."
When going through Section 4 , I think that it is implicitly stated that one has different convergence along different directions.
-----
For this reason, I am not going to change my score.