STABILITY AND CONVERGENCE THEORY FOR LEARNING RESNET: A FULL CHARACTERIZATION
ResNet structure has achieved great success since its debut. In this paper, we study the stability of learning ResNet. Specifically, we consider the ResNet block where is ReLU activation and is a scalar. We show that for standard initialization used in practice, is a sharp value in characterizing the stability of forward/backward process of ResNet, where is the number of residual blocks. Specifically, stability is guaranteed for while conversely forward process explodes when for a positive constant . Moreover, if ResNet is properly over-parameterized, we show for gradient descent is guaranteed to find the global minima footnote{We use to hide logarithmic factor.}, which significantly enlarges the range of that admits global convergence in previous work. We also demonstrate that the over-parameterization requirement of ResNet only weakly depends on the depth, which corroborates the advantage of ResNet over vanilla feedforward network. Empirically, with , deep ResNet can be easily trained even without normalization layer. Moreover, adding can also improve the performance of ResNet with normalization layer.
The authors analyze the stability of a randomly initialized ResNet in terms of the scaling of the network and its depth. They claim to characterize the spectral stability of a randomly initialized network (and local perturbations of it), as well as that of the forward and backward process. These results, along with some further results, are used to extend convergence proofs of ResNets towards more standard initializations. Some of the claims are evaluated empirically. Unfortunately, this paper is not a good contribution at the moment, as the work, although potentially interesting, is somewhat incremental and poorly presented. In particular, the results (including the proofs) only extend the results from Allen-Zhu et al, and are often hard to follow. The presented empirical results are nonetheless interesting and should be expanded upon. Here are some detailed comments - The theorems should be stated more carefully, and in particular, care should be given to make precise which constant terms are hidden away in the big-O and big-Omega notations. E.g. in theorem 2 the probability must have some dependency on c hidden away, which should be made explicit, especially given the assertion “where c can be arbitrarily small”. Similarly, in theorem 1, it is claimed that c is constant, but the theorem is only non-vacuous for either m growing or c ~ log L. There are also some typos in theorem statements (e.g. missing O in theorem 3 and lemma 3), which can easily confuse the reader. - Although I did not have time to check all the proofs, they should be treated with more care. I recommend avoiding big-O and big-Omega notations in the proofs as much as possible, as it can make some steps extremely confusing: in the proof of theorem 1, after “taking an epsilon-net over …”, the probability does not change despite applying an union bound! The union bound argument should be more clearly spelled out, especially given the fact that some smoothness should also be established for this argument (e.g. the trivial sub-multiplicative bound). - The empirical results are interesting, and I think could be taken even further. Indeed, the authors show promising results for learning a global scale parameter (in terms of performance), and it would be interesting to explore: 1) the value of the learned 	au after training: is it still of the order 1 / sqrt{L}? Or does it take some other value. 2) whether it is better to have a 	au per layer or one for the whole network. The presentation of the current results could be improved by including standard deviation when averages are reported, and changing figure 4 with axes starting at zero (bar charts which do not start at zero are extremely misleading). - There are also many typo and grammar issues. Although these do not seriously impede understanding, the paper could be improved by addressing those (at least running a spell-check).
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
STABILITY AND CONVERGENCE THEORY FOR LEARNING RESNET: A FULL CHARACTERIZATION
ResNet structure has achieved great success since its debut. In this paper, we study the stability of learning ResNet. Specifically, we consider the ResNet block where is ReLU activation and is a scalar. We show that for standard initialization used in practice, is a sharp value in characterizing the stability of forward/backward process of ResNet, where is the number of residual blocks. Specifically, stability is guaranteed for while conversely forward process explodes when for a positive constant . Moreover, if ResNet is properly over-parameterized, we show for gradient descent is guaranteed to find the global minima footnote{We use to hide logarithmic factor.}, which significantly enlarges the range of that admits global convergence in previous work. We also demonstrate that the over-parameterization requirement of ResNet only weakly depends on the depth, which corroborates the advantage of ResNet over vanilla feedforward network. Empirically, with , deep ResNet can be easily trained even without normalization layer. Moreover, adding can also improve the performance of ResNet with normalization layer.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The authors analyze the stability of a randomly initialized ResNet in terms of the scaling of the network and its depth. They claim to characterize the spectral stability of a randomly initialized network (and local perturbations of it), as well as that of the forward and backward process. These results, along with some further results, are used to extend convergence proofs of ResNets towards more standard initializations. Some of the claims are evaluated empirically. Unfortunately, this paper is not a good contribution at the moment, as the work, although potentially interesting, is somewhat incremental and poorly presented. In particular, the results (including the proofs) only extend the results from Allen-Zhu et al, and are often hard to follow. The presented empirical results are nonetheless interesting and should be expanded upon. Here are some detailed comments - The theorems should be stated more carefully, and in particular, care should be given to make precise which constant terms are hidden away in the big-O and big-Omega notations. E.g. in theorem 2 the probability must have some dependency on c hidden away, which should be made explicit, especially given the assertion “where c can be arbitrarily small”. Similarly, in theorem 1, it is claimed that c is constant, but the theorem is only non-vacuous for either m growing or c ~ log L. There are also some typos in theorem statements (e.g. missing O in theorem 3 and lemma 3), which can easily confuse the reader. - Although I did not have time to check all the proofs, they should be treated with more care. I recommend avoiding big-O and big-Omega notations in the proofs as much as possible, as it can make some steps extremely confusing: in the proof of theorem 1, after “taking an epsilon-net over …”, the probability does not change despite applying an union bound! The union bound argument should be more clearly spelled out, especially given the fact that some smoothness should also be established for this argument (e.g. the trivial sub-multiplicative bound). - The empirical results are interesting, and I think could be taken even further. Indeed, the authors show promising results for learning a global scale parameter (in terms of performance), and it would be interesting to explore: 1) the value of the learned 	au after training: is it still of the order 1 / sqrt{L}? Or does it take some other value. 2) whether it is better to have a 	au per layer or one for the whole network. The presentation of the current results could be improved by including standard deviation when averages are reported, and changing figure 4 with axes starting at zero (bar charts which do not start at zero are extremely misleading). - There are also many typo and grammar issues. Although these do not seriously impede understanding, the paper could be improved by addressing those (at least running a spell-check).
assistant
The authors analyze the stability of a randomly initialized ResNet in terms of the scaling of the network and its depth. They claim to characterize the spectral stability of a randomly initialized network (and local perturbations of it), as well as that of the forward and backward process. These results, along with some further results, are used to extend convergence proofs of ResNets towards more standard initializations. Some of the claims are evaluated empirically. Unfortunately, this paper is not a good contribution at the moment, as the work, although potentially interesting, is somewhat incremental and poorly presented. In particular, the results (including the proofs) only extend the results from Allen-Zhu et al, and are often hard to follow. The presented empirical results are nonetheless interesting and should be expanded upon. Here are some detailed comments - The theorems should be stated more carefully, and in particular, care should be given to make precise which constant terms are hidden away in the big-O and big-Omega notations. E.g. in theorem 2 the probability must have some dependency on c hidden away, which should be made explicit, especially given the assertion “where c can be arbitrarily small”. Similarly, in theorem 1, it is claimed that c is constant, but the theorem is only non-vacuous for either m growing or c ~ log L. There are also some typos in theorem statements (e.g. missing O in theorem 3 and lemma 3), which can easily confuse the reader. - Although I did not have time to check all the proofs, they should be treated with more care. I recommend avoiding big-O and big-Omega notations in the proofs as much as possible, as it can make some steps extremely confusing: in the proof of theorem 1, after “taking an epsilon-net over …”, the probability does not change despite applying an union bound! The union bound argument should be more clearly spelled out, especially given the fact that some smoothness should also be established for this argument (e.g. the trivial sub-multiplicative bound). - The empirical results are interesting, and I think could be taken even further. Indeed, the authors show promising results for learning a global scale parameter (in terms of performance), and it would be interesting to explore: 1) the value of the learned 	au after training: is it still of the order 1 / sqrt{L}? Or does it take some other value. 2) whether it is better to have a 	au per layer or one for the whole network. The presentation of the current results could be improved by including standard deviation when averages are reported, and changing figure 4 with axes starting at zero (bar charts which do not start at zero are extremely misleading). - There are also many typo and grammar issues. Although these do not seriously impede understanding, the paper could be improved by addressing those (at least running a spell-check).