Neural Tangent Kernel: Convergence and Generalization in Neural Networks
At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit (12; 9), thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function f θ (which maps input vectors to output vectors) follows the kernel gradient of the functional cost (which is convex, in contrast to the parameter cost) w.r.t. a new kernel: the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANNs. While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and it stays constant during training. This makes it possible to study the training of ANNs in function space instead of parameter space. Convergence of the training can then be related to the positive-definiteness of the limiting NTK. We then focus on the setting of least-squares regression and show that in the infinitewidth limit, the network function f θ follows a linear differential equation during training. The convergence is fastest along the largest kernel principal components of the input data with respect to the NTK, hence suggesting a theoretical motivation for early stopping. Finally we study the NTK numerically, observe its behavior for wide networks, and compare it to the infinite-width limit.
The authors prove that networks of infinite width trained with SGD and (infinitely) small step size evolve according to a differential equation, the solution of which depends only on the covariance kernel of the data and, in the case of L2 regression, on the eigenspectrum of the Kernel.  I believe this is a breakthrough result in the field of neural network theory. It elevates the analysis of infinitely wide networks from the study of the static initial function to closely predicting the entire training path.  There are a plethora of powerful consequences about infinitely wide, fully-connected networks:  - They cannot learn information not contained in the covariance matrix - Change to latent representation and parameters tends to zero as width goes to infinity.  - Therefore, disentangled intermediate representations are not learnt - Because the weight change converges to zero, effective depth goes to 2. - The complexity of nonlinearity design, including placing different nonlinearities in different layers, is reduced to NTK design. - If all inputs have the same length, the NTK is a 1d function. Therefore choosing nonlinearities in all layers reduces to choosing a single 1d function. - If all inputs have the same length, assuming a 1-to-1 mapping between NTL and sigma, all networks are equivalent to single-hidden-layer networks.  This doesn't even mention the new analytical tools introduced that will no doubt lead to further important discoveries.  There are some notation / correctness issues.  Consider the term "Gaussian function $f$ of covariance $\Sigma^{(L)}$". In line 122, $f$ is written as a function of a single variable, i.e. $f(x)$ and $f(x')$. But if $f$ is a Gaussian function of one variable, it only has a variance, no covariance. Hence, the statement "$f$ of covariance $\Sigma^{(L)}$" makes no sense.   Further, also on line 122, we have the expression $\mathbb{E}_f$. But if $f$ is a Gaussian function, it is not a random variable, and therefore we cannot take an expectation over it.   Finally, in line 126, we have $f \sim \mathcal{N}(0,\Sigma^{(L-1)})$. Again, this notation suggests that $f$ is a random variable, not a Gaussian function. The notation $\mathcal{N}(0,\Sigma^{(L-1)})$ also suggests that the scalar $\Sigma^{(L-1)}$ is the variance of a 1d random variable, NOT the covariance, as is claimed earlier.  The larger issue is that the notation chosen by the authors suggests that the covariance $\Sigma^{(L)}$ in a given network is only a function of the covariance in the previous layer, $\Sigma^{(L-1)}$. This is not the case. It depends on $\Sigma^{(L-1)}$ and the variance of $\tilde{\alpha}(x)$ in layer L-1 and the variance of $\tilde{\alpha}(x')$ in layer L-1. In fact, one cannot define the covariance $\Sigma^{(L)}$ recursively in isolation. One has to resursively define the triple (variance(x),variance(x'),covariance(x,x')) jointly via recursion. The variances then also show up in the definition of the NTK and pretty much everywhere also the covariance shows up.  The authors should fix this ambiguity in their final version.  Some other minor comments:  - line 63: $\alpha^{(l)}$ is missing the $^$ sign. - in the beginning, $L$ is used to denote the depth of the network and $\ell$ is used for layer recursion. But halfway through, $L$ starts being used for layer recursion. It might be worth changing this for consistency. - in remark 1, it is claimed that under Xavier initialization, weight matrix entries are initialized to independent Gaussians of variance $\frac{1}{n_l}$. In fact, Xavier initialization uses independent centered uniform random variables of variance $\frac{2}{n_l + n_{l+1}}$. Variance $\frac{1}{n_l}$ is sometimes called LeCun initialization. Further, in the paper that introduced Xavier initialization, biases were initialized to zero, not $\mathcal{N}(0,1)$. In fact, initializing biases as $\mathcal{N}(0,1)$ is a terrible choices that is not used, as it would replace a substantial portion of the signal variance with bias in each layer.   ##### Rebuttal response ####  My misunderstanding regarding the covariance / variance issue probably resulted from me having little experience with the common terminology / notation of Kernel machines. I interpreted the recursive formula as trying to establish a recursion for specific values of x,x', i.e. recursing \Sigma^L(x,x') based on the value of \Sigma^L-1(x,x'). However, I see now that you are trying to recurse the entire function \Sigma^L(x,x') as defined for all pairs x,x' based on the entire function of \Sigma^L-1(x,x') for all pairs x,x'. ... That makes sense.  Again, nice paper :)