Published as a conference paper at ICLR 2018 RESIDUAL CONNECTIONS ENCOURAGE ITERATIVE IN- FERENCE
Residual networks (Resnets) have become a prominent architecture in deep learning. However, a comprehensive understanding of Resnets is still a topic of ongoing research. A recent view argues that Resnets perform iterative refinement of features. We attempt to further expose properties of this aspect. To this end, we study Resnets both analytically and empirically. We formalize the notion of iterative refinement in Resnets by showing that residual connections naturally encourage features of residual blocks to move along the negative gradient of loss as we go from one block to the next. In addition, our empirical analysis suggests that Resnets are able to perform both representation learning and iterative refinement. In general, a Resnet block tends to concentrate representation learning behavior in the first few layers while higher layers perform iterative refinement of features. Finally we observe that sharing residual layers naively leads to representation explosion and counterintuitively, overfitting, and we show that simple existing strategies can help alleviating this problem.
This paper shows that residual networks can be viewed as doing a sort of iterative inference, where each layer is trained to use its “nonlinear part” to push its values in the negative direction of the loss gradient.  The authors demonstrate this using a Taylor expansion of a standard residual block first, then follow up with several experiments that corroborate this interpretation of iterative inference.  Overall the strength of this paper is that the main insight is quite interesting — though many people have informally thought of residual networks as having this interpretation — this paper is the first one to my knowledge to explain the intuition in a more precise way.  
Some weaknesses of the paper on the other hand — some of the parts of the paper (e.g. on weight sharing) are only somewhat related to the main topic of the paper. In fact, the authors moved the connection to SGD to the appendix, which I thought would be *more* related.   Additionally, parts of the paper are not as clearly written as they could be and lack rigor.  This includes the mathematical derivation of the main insight — some of the steps should be spelled out more explicitly.  The explanation following is also handwavey despite claims to being formal.   
Some other lower level thoughts:
* Regarding weight sharing for residual layers, I don’t understand why we can draw the conclusion that the initial gradient explosion is responsible for the lower generalization capability of the model with shared weights.  Are there other papers in literature that have shown this connection?
* The name “cosine loss” suggests that this function is actually being minimized by a training procedure, but it is just a value that is being plotted… perhaps just call it the cosine?
* I recommend that the authors also check out Figurnov et al CVPR 2017 ("Spatially Adaptive Computation Time for Residual Networks") which proposes an “adaptive” version of ResNet based on the intuition of adaptive inference.
* The plots in the later parts of the paper are quite small and hard to read.  They are also spaced together too tightly (horizontally), making it difficult to immediately see what each plot is supposed to represent via the y-axis label.
* Finally, the citations need to be fixed (use \citep{} instead of \cite{})