Residual Networks Behave Like Ensembles of Relatively Shallow Networks
In this work we propose a novel interpretation of residual networks showing that they can be seen as a collection of many paths of differing length. Moreover, residual networks seem to enable very deep networks by leveraging only the short paths during training. To support this observation, we rewrite residual networks as an explicit collection of paths. Unlike traditional models, paths through residual networks vary in length. Further, a lesion study reveals that these paths show ensemble-like behavior in the sense that they do not strongly depend on each other. Finally, and most surprising, most paths are shorter than one might expect, and only the short paths are needed during training, as longer paths do not contribute any gradient. For example, most of the gradient in a residual network with 110 layers comes from paths that are only 10-34 layers deep. Our results reveal one of the key characteristics that seem to enable the training of very deep networks: Residual networks avoid the vanishing gradient problem by introducing short paths which can carry gradient throughout the extent of very deep networks.
The authors give a novel interpretation of deep residual networks, a feed-forward architecture that has received significant attention from the research community recently. They state that residual networks are ensembles of relatively shallow networks. They provide detailed experiments to underpin their interpretation.
The paper approaches an important and interesting question with a novel methodology. It's an inspiring, thought-provoking read, especially for those of us interested in residual networks. It provokes us into looking at resnets (and deep nets in general) from a different perspective. The single big issue I see with it is that its claims are either not formalized enough, and inasmuch as they are formalized, they are not sufficiently justified by the (admittedly extemely well-thought-out and informative) experiments presented. Below are a couple of examples to make this critique a bit more specific.  lines 157-158: Here the authors argue for the ensemble view by invoking the observation that single blocks can be removed from the network without significant performance degradation. I claim that this line of argument is not logically sound. A simpler explanation of the robustness phenomenon is that the magnitude of individual residual outputs is small compared to their summed magnitude. (I've actually conducted experiments on deep residual networks that verify this.) Of course, one could use the word "ensemble" in such a general sense that this alternative explanation (of small individual resnet block magnitudes) is not in fact an alternative explanation, but rather, it supports the ensemble claim. I'd say that in this case the ensemble claim is too vacuous. In any case, it's never formalized enough so that the reader can decide.  One other way of highlighting the IMHO problematic line of argument is this: A particularly  useless kind of ensemble is the echo chamber, where the individual elements of the ensemble all vote the same way. I believe that the arguments and experiments presented in the paper never successfully exclude the possibility that the observed ensemble is of this kind.  A related observation: In principle, adding a constant vector "drift" to the output of a residual block does not hurt the learning capabilities of the system: the following blocks simply must learn to work on a translated version of their original input domain. But -- unless those blocks learn to operate both on the translated and the original domain, an a priori unexpected form of generalization -- the artificial removal of the residual block will damage the system, because it removes the drift as well. The experiments presented in the paper clearly show that this degradation does not actually happen: there's no significant drift. It would be interesting to understand this phenomenon better. Is it related to the use of ReLUs or batchnorm? Or is it something more general? I have a vague general issue with the intuition behind the "unraveled view" of the network. The easiest way to state this is through a crude analogy: We can look at the "unraveled view" of the multilinear formula (1+x_1)(1+x_2)...(1+x_n): a polynomial in standard form, with 2^n tems. It's hard to imagine what the unraveled view adds to the discussion in this specific case. When compared to the 1-multiplicity formula x_1x_2...x_n, there are very few meaningful senses of the word in which our 2^n-multiplicity multilinear formula is exponentially more complex. I'm aware that multiplication and function composition are mixed here. It's an imperfect analogy. Still, I would recommend the authors to defuse similar kinds of objections in advance.  The above criticisms should not obscure the fact that I've found the paper thought-provoking, the experiments well-designed, and the potential impact high.