A recent line of work studies overparametrized neural networks in the "kernel regime," i.e. when the network behaves during training as a kernelized linear predictor, and thus training with gradient descent has the effect of finding the minimum RKHS norm solution. This stands in contrast to other studies which demonstrate how gradient descent on overparametrized multilayer networks can induce rich implicit biases that are not RKHS norms. Building on an observation by Chizat and Bach (2018), we show how the scale of the initialization controls the transition between the "kernel" (aka lazy) and "rich" (aka active) regimes and affects generalization properties in multilayer homogeneous models. We provide a complete and detailed analysis for a simple two-layer model that already exhibits an interesting and meaningful transition between the kernel and rich regimes, and we demonstrate the transition for more complex matrix factorization models and multilayer non-linear networks.
This paper analyzes an inductive bias of the gradient flow for diagonal two-or higher-homogeneous models and characterizes a limit point depending on the initialization scale of parameters. Concretely, the paper shows that the gradient flow converges to an interpolator attaining minimum L1- (or L2-norm) when the scale is small (or large). In addition, these analyses are well verified empirically on MNIST and CIFAR-10 datasets.
Quality:
The work is of good quality and is technically sound.
Clarity:
The paper is well organized and easy to read.
Significance:
To explain the generalization ability of powerful machine learning models that can perfectly learn a training dataset, the implicit bias of the optimization methods and models play key roles when explicit regularization is not adopted. For instance, deep neural networks fall into this scenario. I think this paper makes a better contribution in this line of researches. Although, homogeneous models treated in this study is restricted (essentially linear models) and a theory is limited to the continuous gradient flow, these settings are rather common in this context. In [Gunasekar+(2017)], the convergence to the minimum L1-norm solution was shown for a slightly different model when the scale goes to zero. However, in addition to this property, the paper analyzes arbitrary scales of parameters and shows the convergence to the minimum L2-norm solution when the scale goes to infinity for diagonal homogeneous models.
It would be nice if the authors could emphasize the technical difficulty compared to [Gunasekar+(2017)] to strengthen the contribution of the paper.
A few questions:
- Can this analysis be extended to the setting of early stopping? Toward a better explanation of the generalization performance of deep learning, understanding of the inductive bias of the early stopping before convergence is more important.   
- A provided theory is limited to linear models essentially. Is it possible to extend a theory to non-linear models?
-----
Update:
I thank the authors for the response. I am convinced of the difference from [Gunasekar+(2017)] and my review stands. I  would like to keep my score.