Empirical evidence suggests that neural networks with ReLU activations generalize better with over-parameterization. However, there is currently no theoretical analysis that explains this observation. In this work, we study a simplified learning task with over-parameterized convolutional networks that empirically exhibits the same qualitative phenomenon. For this setting, we provide a theoretical analysis of the optimization and generalization performance of gradient descent. Specifically, we prove data-dependent sample complexity bounds which show that overparameterization improves the generalization performance of gradient descent.
The paper studies a particular task (the XOR detection problem) in a particular setup (see below), and proves mathematically that in that case, the training performs better when the number of features grows.
The task is the following one:
- consider a set of pairs of binary values (-1 or +1);
- detect whether at least one of these pairs is (+1, +1) or (-1, -1).
The design of the predictor is:
- for each pair, compute 2k features (of the form ReLu(linear combination of the values, without bias));
- compute the max over all pairs of these features (thus obtaining 2k values);
- return the k first values minus the k last ones.
The training set consists only of examples having the following property [named 'diversity']:
- if the example (which is a set of pairs) is negative (i.e. doesn't contain (+1,+1) nor (-1,-1)), then it contains both (-1,1) and (1,-1);
- if the example is positive, it contains all possible pairs.
The paper proves that, under this setup, training with a number of features k > 120 will perform better than with k = 2 only (while k = 2 is theoretically sufficient to solve the problem). While tackling an interesting problem (impact of over-parameterization), the proof is specific to this particular, unusual architecture, with a "max - max" over features independently computed for each pair that the example contains; it relies heavily on the fact that the input are binary, and that the number of possible input pairs is small (4), which implies that the features can take only 4 values. Note also that the probabilities in some theorems are not really probabilities of convergence/performance of the training algorithm per se (as one would expect in such PAC-looking bounds), but actually probabilities of the batch of examples to all satisfy some property (the diversity).
Thus it is difficult to get from this study any insight about the over-parameterization / training ability phenomenon, for more general tasks, datasets or architectures.
Though clearly an impressive amount of work has been done in this proof, I do not see how it can be generalized (there is no explanation in the paper in that regard either, while it would have been welcomed), and consequently be of interest for the vast majority of the ICLR community, which is why I call for rejection.