Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
Neural networks have many successful applications, while much less theoretical understanding has been gained. Towards bridging this gap, we study the problem of learning a two-layer overparameterized ReLU neural network for multi-class classification via stochastic gradient descent (SGD) from random initialization. In the overparameterized setting, when the data comes from mixtures of well-separated distributions, we prove that SGD learns a network with a small generalization error, albeit the network has enough capacity to fit arbitrary labels. Furthermore, the analysis provides interesting insights into several aspects of learning neural networks and can be verified based on empirical studies on synthetic data and on the MNIST dataset.
This paper analyzes the generalization performance of two-layer ReLU networks in an overparametrized regime. Specifically, the networks are trained with SGD over separable multiclass  problems. In such setting, the authors prove that the learned network has small generalization error with high probability. This result generalizes the result in [7], which only considers linearly separable data.   This paper is well organized and easy to follow. The proof sketch and discussion on insights make it easy to understand the technique in use and the implications of this work.   There is no doubt that the authors address an important problem. Explaining the generalization performance of deep learning is a very core task of the modern machine learning. But the contribution of this work seems incremental. The theoretical result of this paper generalizes the result in [7] by adopting a more general separable dataset. It is true that such structured (multiclass) data is closer to the practical scenario. While the truly practical scenarios normally involve non-separable dataset, which cannot be easily analyzed with the approach used in the paper (the separability parameter \delta > 0 seems necessary even for the simplified case in Section 5).   Moreover, mixing the generalization and optimization in the main theorem likely complicates the analysis and restricts the scope of the result. In particular, SGD might have nothing special in regard to learning a solution with good generalization performance. It just happens to be the only practical method for neural network training. For random forest [a] and kernel machine [b], solutions that are not learned by SGD (e.g. exact solution for kernel regression) can achieve both (almost) zero training error and excellent generalization performance. Note that all these models/architectures are overparametrized. So we see empirically that for several overparametrized architectures, methods other than SGD can learn a solution with a small generalization error. Thus, it is hard to tell which inductive bias is more important to obtain the good generalization performance, the one derived from the optimization method or the one enforced by the architecture selection.    Some minor things:   Line 123: or every \epsilon ---> for every \epsilon.   Line 141: by SGD ---> by GD.   Line 276: converges to 0 ---> converges to 100%.   [a] Wyner, Abraham J., et al. "Explaining the success of adaboost and random forests as interpolating classifiers."  [b] Belkin, Mikhail, et al. "To understand deep learning we need to understand kernel learning."   ===== After Rebuttal ===== After reading the rebuttal and all reviews, I feel that this paper deserves a higher score.