Most modern learning problems are highly overparameterized, meaning that the model has many more parameters than the number of training data points, and as a result, the training loss may have infinitely many global minima (in fact, a manifold of parameter vectors that perfectly interpolates the training data). Therefore, it is important to understand which interpolating solutions we converge to, how they depend on the initialization point and the learning algorithm, and whether they lead to different generalization performances. In this paper, we study these questions for the family of stochastic mirror descent (SMD) algorithms, of which the popular stochastic gradient descent (SGD) is a special case. Recently it has been shown that, for overparameterized linear models, SMD converges to the global minimum that is "closest" (in terms of the Bregman divergence of the mirror used) to the initialization point, a phenomenon referred to as implicit regularization. Our contributions in this paper are both theoretical and experimental. On the theory side, we show that in the overparameterized nonlinear setting, if the initialization is close enough to the manifold of global optima, SMD with sufficiently small step size converges to a global minimum that is approximately the closest global minimum in Bregman divergence, thus attaining approximate implicit regularization. For highly overparametrized models, this closeness comes for free: the manifold of global optima is so high dimensional that with high probability an arbitrarily chosen initialization will be close to the manifold. On the experimental side, our extensive experiments on the MNIST and CIFAR-10 datasets, using various initializations, various mirror descents, and various Bregman divergences, consistently confirms that this phenomenon indeed happens in deep learning: SMD converges to the closest global optimum to the initialization point in the Bregman divergence of the mirror used. Our experiments further indicate that there is a clear difference in the generalization performance of the solutions obtained from different SMD algorithms. Experimenting on the CIFAR-10 dataset with different regularizers, 1 to encourage sparsity, 2 (yielding SGD) to encourage small Euclidean norm, and 10 to discourage large components in the parameter vector, consistently and definitively shows that, for small initialization vectors, 10 -SMD has better generalization performance than SGD, which in turn has better generalization performance than 1 -SMD. This surprising, and perhaps counter-intuitive, result strongly suggests the importance of a comprehensive study of the role of regularization, and the choice of the best regularizer, to improve the generalization performance of deep networks.
This paper studies the performance of the mirror gradient method when applied to the overparameterized network. The authors claim that the SMD method could find the regularized global minimize for different potential functions, in terms of minimal Bregman distance. Further experiments are carried out to back up the authors' claim.
However, the drawbacks of this paper are listed in these several points:
1) The regularization w.r.t. Bregman distance is quite different from the regularization used in network training: instead of $\|w - w_0\|$, we use $\|w\|$ more often, therefore, the virtue of sparsity shared by 1-norm is not exploited.
2) The authors' assumption in Assumption 3.1 is too fuzzy: instead of detailed analysis in the papers related to the overparameterized network, the author does not give ANY relationship between the $\epsilon$ and these three important parameters: a) network width, b) probability introduced by random initialization, c) the number of input data.
3) Therefore, according to the too strong and fuzzy assumption mentioned in the last point, Assumption 3.1 simply makes the network equals to the linear model, which leads to minor contributions according to the study of the overparameterized network.
4) The `over parameterized network' discussed now, including the work of (Li & Liang, 2018; Du et al., 2018; Azizan & Hassibi, 2019; Allen-Zhu et al., 2019; Cao & Gu, 2019), are mainly focused on the `network of infinite width', however, in the authors' experiment, the network architecture, e.g. ResNet18, is more to be a `very deep network' rather than a `very wide network'. Therefore, the authors' theory is built on Assumption 3.1, which is based on `network of infinite width', while the experiment is built on the `very deep network'. The result of the experiments is not enough to support the authors' theorem.
In conclusion, I am convinced that the authors' work is over-claimed in this paper.