ResNet structure has achieved great success since its debut. In this paper, we study the stability of learning ResNet. Specifically, we consider the ResNet block h l = φ(h l−1 + τ · g(h l−1 )) where φ(·) is ReLU activation and τ is a scalar. We show that for standard initialization used in practice, τ = 1/Ω( √ L) is a sharp value in characterizing the stability of forward/backward process of ResNet, where L is the number of residual blocks. Specifically, stability is guaranteed for τ ≤ 1/Ω( √ L) while conversely forward process explodes when τ > L − 1 2 +c for a positive constant c. Moreover, if ResNet is properly over-parameterized, we show for τ ≤ 1/Ω( √ L) gradient descent is guaranteed to find the global minima 1 , which significantly enlarges the range of τ ≤ 1/Ω(L) that admits global convergence in previous work. We also demonstrate that the over-parameterization requirement of ResNet only weakly depends on the depth, which corroborates the advantage of ResNet over vanilla feedforward network. Empirically, with τ ≤ 1/ √ L, deep ResNet can be easily trained even without normalization layer. Moreover, adding τ = 1/ √ L can also improve the performance of ResNet with normalization layer.
This paper shows the stability of ResNet for the output scale of resblock: $\tau < \sqrt{L}^{-1}$ and shows the explosion for $\tau > L^{-0.5+c}$. Based on this analysis, a linear convergence rate of the gradient descent for the squared loss is also shown. In addition, this paper empirically verifies the efficiency of the initialization of $\tau = \sqrt{L}^{-1}$ with/without batch normalization.
Contributions of the paper are summarized below:
- Provides a sharp characterization of the largest scale of outputs of resblocks: $\tau$.
- Enlarges the class of ResNets with the global convergence guarantee of the gradient descent w.r.t. the scale of $\tau$. (A related study [Allen-Zhu+2018b] focused on the scale of $\tau = 1/L$).
- Improve the depth dependence in the network width and the iteration complexity compared to existing studies.
Significance:
To theoretically justify the success of ResNets, this line of research is important, and I think this paper makes a certain contribution in this sense, although there seems to be much room for improvements regarding the depth dependence in the network width. (Unrealistically high over-parameterization: $m=n^8L^7$ is still required). In addition, a proposed technique (Theorem 1) to derive gradient bounds is surprising and beyond the intuition. Indeed, a simple way of using a natural spectral bound cannot explain the stability of ResNets with $\tau=L^{-0.5}$.
Clarity:
The paper is well organized but there are some technical concerns as described below.
Quality:
I think the quality of the paper can be improved. 
- Maybe probabilities in some statements are not correct, for instance, a probability in Theorem 2 should depend on the value of $c$.
- It is better to specify the value of $c$ in Theorem 1. It may take an arbitrary small value at the expense of the probability.
- In the convergence analysis of the gradient descent (Theorem 5), the dependency on $\tau$ should be mentioned. Is there no effect on the convergence rate by the choice of $\tau$?
Minor comments:
- The network architecture is different from the standard ResNets. Usually, non-linear activation functions are applied after resblocks. Can theoretical analyses be extended to such a setting?
- Initialization scales of parameters in related studies, as well as $\tau$, should also be clarified for a fair comparison of $\tau$.
- In proofs, citations of theorems and lemmas seem incorrect, e.g., Theorem 1 may be used in the proof of Lemma 3 (and Theorem 3 too?) instead of Lemma 1.
- Typo In Theorem 1: for all $a > l$  --> for all $b > a$ (?).
- There is no definition of $h'_{i,l}$ in Lemma 4. Is it a perturbation of a hidden node h?
-----
Update: 
I thank the authors for the response and hard work.
I would like to keep my score. Some concerns have been resolved. Moreover,  I think highly of a proof technique (Theorem 1). However, the advantage of ResNets is still unclear because high over-parameterization is required.