Are ResNets Provably Better than Linear Predictors?
A residual network (or ResNet) is a standard deep neural net architecture, with stateof-the-art performance across numerous applications. The main premise of ResNets is that they allow the training of each layer to focus on fitting just the residual of the previous layer's output and the target output. Thus, we should expect that the trained network is no worse than what we can obtain if we remove the residual layers and train a shallower network instead. However, due to the non-convexity of the optimization problem, it is not at all clear that ResNets indeed achieve this behavior, rather than getting stuck at some arbitrarily poor local minimum. In this paper, we rigorously prove that arbitrarily deep, nonlinear residual units indeed exhibit this behavior, in the sense that the optimization landscape contains no local minima with value above what can be obtained with a linear predictor (namely a 1-layer network). Notably, we show this under minimal or no assumptions on the precise network architecture, data distribution, or loss function used. We also provide a quantitative analysis of approximate stationary points for this problem. Finally, we show that with a certain tweak to the architecture, training the network with standard stochastic gradient descent achieves an objective value close or better than any linear predictor.
Summary This paper studies loss surface of residual networks (ResNets). The main contribution of this paper is to prove that any local minimum of risk (empirical or population) of ResNet is better than the risk achieved by the best linear predictor. The authors prove this by upper-bounding the gap between the risk achieved by ResNet and the best linear predictor for “approximate local minima,” which is defined as \epsilon-SOPSP in the paper. For any local minima of ResNet, the upper bound becomes zero, which implies that the local minima is at least as good as (in terms of risk value) the best linear predictor.   After that the authors point out that, the bound on the gap depends on the norm of parameters, so the bound is small only if the parameters are close to zero. Thus, unless one can bound the norm of parameters, no meaningful algorithmic result can be derived from the main theorem. They show by an example in which an approximate local minimum has much worse risk than linear predictor, and also that regularization can introduce spurious local minimum.  The paper concludes with an algorithmic result that, by tweaking the ResNet architecture, training with stochastic gradient descent achieves a risk value close to (or better than) linear predictors.  Comments I like this work a lot. Understanding the loss surface of neural network models is very important, and this theoretical work proves that ANY local minima of ResNet are provably better than linear models. Given that many other existing results require differentiability of local minima, it is very interesting that the main theorem works even for non-differentiable local minima. It is also impressive that they prove this under almost no assumption on data distribution, network architecture, or loss function.  The authors are honest about the weakness of the main theorem; they admit that it is hard to get an algorithmic result. However, they partially remedy this issue by tweaking the architecture with almost the same expressive power and show a positive algorithmic theorem.  The writing of the paper is very clear and the paper reads smoothly.   Overall, it is a well-written paper that contains an interesting result on the energy landscape of ResNets. I recommend acceptance.  Minor points - In the proof sketch of Lemma 1, I think it’d be better to be more explicit about the definition of $d\ell$, for example: $d\ell = \frac{\partial}{\partial p} \ell(p;y) \vert_{p = w^T (x+V f_\theta (x))}$. And maybe replace $d\ell$ with some other single character?  - There is a missing right parenthesis in the beginning of Theorem 2.  ***************Modified on 7/31*************** I’m terribly sorry that it’s already late, but I want to ask the authors (if they have chance to see this) for clarification of whether the ResNet architecture in the paper can be implemented with a stack of standard ResNet units.  In the “pre-activation” structure, usually each resnet unit is defined as $t \mapsto t + g_i(t)$, where $g_i(t)$ can be some network, say $g_i (t) = V_i * ReLU(U_i * t)$. If we stack $L > 1$ such residual units, the network would look like  h_0 = x, h_i = h_{i-1} + g_i( h_{i-1} ) for i = 1, … , L, h_{L+1} = w^T * h_L,  and the output $h_{L+1}$ can be written as  h_{L+1}  = w^T [ x + g_1(x) + g_2(h_1) + … + g_L(h_{L-1}) ] = w^T [ x + V_1 * ReLU(U_1 * x) + V_2 * ReLU(U_2 * h_1) + … + V_L * ReLU(U_L * h_{L-1}) ].  In contrast, in this paper, the authors analyze a ResNet $x \mapsto w^T [ x + V f_\theta (x) ]$. To me, it seems difficult to formulate the above network output $h_{L+1}$ into this particular form.  Does the theorem in the paper hold for this arbitrarily deep stack of residual units, or does it only hold for *only one* residual unit followed by a linear layer? If the former is the case, can you clarify why it holds for $L>1$ units? If the latter is the case, the paper should in some way emphasize this, because readers can get confused between “$f_\theta$ can have arbitrary structure and depth” and “the entire ResNet can be arbitrarily deep.” A possible option could be to refer to the architecture as a “shallow ResNet.”  Again, I’m so sorry that I’m too late for additional questions and comments. If you get to see this modification before the deadline, it would be very helpful if you can provide answers.  ***************Modified after rebuttal*************** I would like to thank the authors for generously providing feedback to my last-minute questions. And yes, it’d be more helpful for readers to emphasize (although the authors already put significant efforts on it!) the difference between a residual unit vs network, because some readers may not notice the difference of the terms (unit vs network) and then misunderstand the point of the paper; actually, this almost happened to me.  Even though the paper only considers ResNets with a single residual unit, my evaluation of the paper stays the same. This is because this paper pushes forward the understanding of loss surface of ResNets, and many existing theoretical works on (conventional) fully connected neural nets are also limited to one-hidden-layer networks.