Recent research shows that the following two models are equivalent: (a) infinitely wide neural networks (NNs) trained under 2 loss by gradient descent with infinitesimally small learning rate (b) kernel regression with respect to so-called Neural Tangent Kernels (NTKs) (Jacot et al., 2018). An efficient algorithm to compute the NTK, as well as its convolutional counterparts, appears in Arora et al. (2019a), which allowed studying performance of infinitely wide nets on datasets like CIFAR-10. However, super-quadratic running time of kernel methods makes them best suited for small-data tasks. We report results suggesting neural tangent kernels perform strongly on low-data tasks.
This paper conducts very interesting and meaningful study of kernels induced by infinitely wide neural networks on small data tasks. They show that on a variety of tasks performance of these kernels are superior to both finite neural networks and Random Forest methods.
While neural tangent kernel (NTK) [1] is motivated for studying training dynamics of neural networks, it is also important to ask to find utility of these new powerful kernels that captures functional priors of neural networks. This paper conducted important study on small dataset regime and on a wide range of tasks (90 UCI datasets, small subset of CIFAR-10, few shot image classification task on VOC07. 
Authors introduce a family of generalized NTK kernels interpolating between NNGP kernels [2] to original NTK[1] by fixing first L’ layers and allowing to train remaining layers. Treating L’ as a hyperparameter, the authors try both NNGP/NTK and kernels in between as well. 
Another contribution I observe is applying kernel SVM where one utilizes NTK and shows that it can work well. This paper shows that kernels induced by infinitely wide networks could become useful for real world applications where data size is not so large. 
There are few small concerns regarding experiments which are discussed in detailed comments. Overall I think the message of the paper is clear and well supported therefore I recommend accepting the paper. 
 
Detailed comments
	
1) From reading the paper it was not easy to grasp where point 4 of the abstract was based on. 
2) In the first footnote, small nit is that, in practice one should not invert matrix but just do a linear solve for better numerical stability and efficiency (still O(N^3) but with better constant)
3) In section 3, there seems to be no bias. Are NTK and NNs considered in this work contain no bias? Or is bias ignored for ease of presentation? 
4) Nit p4 first paragraph in section 4 : multiplayer -> multilayer
5) Regards to NTK initialization performing better than standard He initialization: It was observed in [3] that for multilayer perceptron both parameterization is on-par but for CNN or WideResNet case standard parameterization performed significantly better.
6) Note that similar to analysis in section 5,  for CIFAR-10 with fully connected model [1] shows that for all dataset size(100-45k) NNGP performs better than trained neural networks.
7) One may worry that ResNet-34 is not properly tuned as most hyperparameters were fixed for large dataset. 
8) Regards to hyperparameters for NTK, is there a consistent trend one could find regards to L’? What percentage of tasks that NTK performed well actually have a high L’?
9) To help the readers, I would suggest adding a little more description on statistics used for comparison as well as what VOC07 task entails. 
[1] Jacot et al., Neural Tangent Kernel: Convergence and Generalization in Neural Networks, NeurIPS 2018
[2] Lee et al., Deep Neural Networks as Gaussian Processes, ICLR 2018
[3] Park et al., The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study, ICML 2019
EDIT AFTER AUTHOR RESPONSE:
I have read the response from authors. I appreciate all the efforts to improve the paper.