The Effect of Residual Architecture on the Per-Layer Gradient of Deep Networks
A critical part of the training process of neural networks takes place in the very first gradient steps post initialization. In this work, we study the connection between the network*s architecture and initialization parameters, to the statistical properties of the gradient in random fully connected ReLU networks, through the study of the the Jacobian. We compare three types of architectures: vanilla networks, ResNets and DenseNets. The later two, as we show, preserve the variance of the gradient norm through arbitrary depths when initialized properly, which prevents exploding or decaying gradients at deeper layers. In addition, we show that the statistics of the per layer gradient norm is a function of the architecture and the layer*s size, but surprisingly not the layer*s depth. This depth invariant result is surprising in light of the literature results that state that the norm of the layer*s activations grows exponentially with the specific layer*s depth. Experimental support is given in order to validate our theoretical results and to reintroduce concatenated ReLU blocks, which, as we show, present better initialization properties than ReLU blocks in the case of fully connected networks.
This paper studies the effects of residual and dense net type connections on the moments of per-layer gradients at random initialization. In particular, using duality, bounds on the variance of the square norm of Jacobian (with respect to the randomness of random initialization) are derived for vanilla networks. By noticing that connections that bypass a specific layer does not affect the expected square norm of the gradients of a particular layer, the paper continue to characterize the bounds for residual networks and densely connected networks that have special skip connections. In particular, with properly chosen initialization scales for each layer, the architectures with skip connections can be initialized so that the gradient norm does not explode with increasing depth. In the second part of the paper, the topic shifted to a bit unrelated topic, comparing the effects of ReLU vs concatenated ReLU blocks on the norm of layer activations. The results in this paper seem to be useful for guidance on choosing proper initialization schemes for different architectures at different depths. I hope the paper could spend more efforts discussing the potential impact of their results. Moreover, some preliminary empirical studies could also help: - how well do the bounds still hold after the optimization starts to run? - does the proposed initialization scheme actually allow training of very deep neural networks? Minor comments: - please add x and y labels to the plots - please add grid (as in Fig 4) to Fig 2 and 3. Currently it is especially hard to decide the slope for curves in Fig 2b.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
The Effect of Residual Architecture on the Per-Layer Gradient of Deep Networks
A critical part of the training process of neural networks takes place in the very first gradient steps post initialization. In this work, we study the connection between the network*s architecture and initialization parameters, to the statistical properties of the gradient in random fully connected ReLU networks, through the study of the the Jacobian. We compare three types of architectures: vanilla networks, ResNets and DenseNets. The later two, as we show, preserve the variance of the gradient norm through arbitrary depths when initialized properly, which prevents exploding or decaying gradients at deeper layers. In addition, we show that the statistics of the per layer gradient norm is a function of the architecture and the layer*s size, but surprisingly not the layer*s depth. This depth invariant result is surprising in light of the literature results that state that the norm of the layer*s activations grows exponentially with the specific layer*s depth. Experimental support is given in order to validate our theoretical results and to reintroduce concatenated ReLU blocks, which, as we show, present better initialization properties than ReLU blocks in the case of fully connected networks.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper studies the effects of residual and dense net type connections on the moments of per-layer gradients at random initialization. In particular, using duality, bounds on the variance of the square norm of Jacobian (with respect to the randomness of random initialization) are derived for vanilla networks. By noticing that connections that bypass a specific layer does not affect the expected square norm of the gradients of a particular layer, the paper continue to characterize the bounds for residual networks and densely connected networks that have special skip connections. In particular, with properly chosen initialization scales for each layer, the architectures with skip connections can be initialized so that the gradient norm does not explode with increasing depth. In the second part of the paper, the topic shifted to a bit unrelated topic, comparing the effects of ReLU vs concatenated ReLU blocks on the norm of layer activations. The results in this paper seem to be useful for guidance on choosing proper initialization schemes for different architectures at different depths. I hope the paper could spend more efforts discussing the potential impact of their results. Moreover, some preliminary empirical studies could also help: - how well do the bounds still hold after the optimization starts to run? - does the proposed initialization scheme actually allow training of very deep neural networks? Minor comments: - please add x and y labels to the plots - please add grid (as in Fig 4) to Fig 2 and 3. Currently it is especially hard to decide the slope for curves in Fig 2b.
assistant
This paper studies the effects of residual and dense net type connections on the moments of per-layer gradients at random initialization. In particular, using duality, bounds on the variance of the square norm of Jacobian (with respect to the randomness of random initialization) are derived for vanilla networks. By noticing that connections that bypass a specific layer does not affect the expected square norm of the gradients of a particular layer, the paper continue to characterize the bounds for residual networks and densely connected networks that have special skip connections. In particular, with properly chosen initialization scales for each layer, the architectures with skip connections can be initialized so that the gradient norm does not explode with increasing depth. In the second part of the paper, the topic shifted to a bit unrelated topic, comparing the effects of ReLU vs concatenated ReLU blocks on the norm of layer activations. The results in this paper seem to be useful for guidance on choosing proper initialization schemes for different architectures at different depths. I hope the paper could spend more efforts discussing the potential impact of their results. Moreover, some preliminary empirical studies could also help: - how well do the bounds still hold after the optimization starts to run? - does the proposed initialization scheme actually allow training of very deep neural networks? Minor comments: - please add x and y labels to the plots - please add grid (as in Fig 4) to Fig 2 and 3. Currently it is especially hard to decide the slope for curves in Fig 2b.