Answers.Body: <p><em>Before proceeding, it's important to note that ResNets, as pointed out <a href="https://ai.stackexchange.com/q/18703/2444">here</a>, were not introduced to <em>specifically</em> solve the VGP, but to improve learning in general. In fact, the authors of ResNet, in the original paper, noticed that neural networks without residual connections don't learn as well as ResNets, although they are using batch normalization, which, in theory, ensures that gradients should not vanish (<a href="https://arxiv.org/pdf/1512.03385.pdf#page=5" rel="nofollow noreferrer">section 4.1</a>). So, in this answer, I'm just giving a potential explanation of why ResNets <strong>may also</strong> mitigate (or prevent to some extent) the VGP, but the cited research papers below also confirm that ResNets prevent the VGP. Given that I didn't fully read all the papers mentioned in this answer, the information in this answer may not be fully accurate.</em></p>
  <p>The skip connections allow information to skip layers, so, in the forward pass, information from layer <span class="math-container">$l$</span> can directly be fed into layer <span class="math-container">$l+t$</span> (i.e. the activations of layer <span class="math-container">$l$</span> are added to the activations of layer <span class="math-container">$l+t$</span>), for <span class="math-container">$t \geq 2$</span>, and, during the forward pass, the gradients can also flow unchanged from layer <span class="math-container">$l+t$</span> to layer <span class="math-container">$l$</span>.</p>
  <p>How exactly could this prevent the vanishing gradient problem (VGP)? The VGP occurs when the elements of the gradient (the partial derivatives with respect to the parameters of the NN) become exponentially small, so that the update of the parameters with the gradient becomes almost insignificant (i.e. if you add a very small number <span class="math-container">$0 &lt; \epsilon \ll 1$</span> to another number <span class="math-container">$d$</span>, <span class="math-container">$d+\epsilon$</span> is almost the same as <span class="math-container">$d$</span>) and, consequently, the NN learns very slowly or not at all (considering also numerical errors). Given that these partial derivatives are computed with the <a href="https://en.wikipedia.org/wiki/Chain_rule" rel="nofollow noreferrer">chain rule</a>, this can easily occur, because you keep on multiplying small (finite-precision) numbers (please, have a look at how the chain rule works, if you're not familiar with it). For example, <span class="math-container">$\frac{1}{5}\frac{1}{5} = \frac{1}{25}$</span> and then <span class="math-container">$\frac{1}{5}\frac{1}{25} = \frac{1}{125}$</span>, and so on. The deeper the NN, the more likely the VGP can occur. This should be quite intuitive if you are familiar with the chain rule and the back-propagation algorithm (i.e. the chain rule). By allowing information to skip layers, layer <span class="math-container">$l+t$</span> receives information from both layer <span class="math-container">$l+t-1$</span> and layer <span class="math-container">$l$</span> (unchanged, i.e. you do not perform multiplications). For example, to compute the activation of layer <span class="math-container">$l+t-1$</span>, you perform the usual linear combination followed by the application of the non-linear activation function (e.g. ReLU). In this linear combination, you perform multiplications between numbers that could already be quite small, so the results of these multiplications are even smaller numbers. If you use <a href="https://stats.stackexchange.com/q/174295/82135">saturating activation functions (e.g. tanh)</a>, this problem can even be aggravated. If the activation of layer <span class="math-container">$l+t$</span> are even smaller than the activations of layer <span class="math-container">$l+t-1$</span>, the addition of the information from layer <span class="math-container">$l$</span> will make these activations bigger, thus, to some extent, they will prevent these activations from becoming exponentially small. A similar thing can be said for the back-propagation of the gradient.</p>
  <p>Therefore, skip connections can mitigate the VGP, and so they can be used to train deeper NNs.</p>
  <p>These explanations are roughly consistent with the findings reported in the paper <a href="https://arxiv.org/pdf/1605.06431.pdf" rel="nofollow noreferrer">Residual Networks Behave Like Ensembles of Relatively Shallow Networks</a>, which states</p>
  <blockquote>
  <p>Our results reveal one of the key characteristics that seem to enable the training of very deep networks: Residual networks avoid the vanishing gradient problem by introducing <strong>short paths</strong> which can carry gradient throughout the extent of very deep networks.</p>
  </blockquote>
  <p>In the paper <a href="https://arxiv.org/pdf/1805.07477.pdf" rel="nofollow noreferrer">Norm-Preservation: Why Residual Networks Can Become Extremely Deep?</a>, the authors also discuss another desirable effect of skip connections.</p>
  <blockquote>
  <p>We show theoretically and empirically that each residual block in ResNets is increasingly <strong>norm-preserving</strong>, as the network becomes deeper</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-03T17:44:27.420Z
Answers.Text: Thank you, as I can see, it is not ReLU that do the main job, but the skip connections
Answers.UserDisplayName: FraMan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-03T17:51:51.713Z
Answers.Text: @FraMan I would say that both contribute to mitigating the problem. The ReLU, as opposed to e.g. the sigmoid, does not squash its inputs. It basically only sets negative values to zero and lets everything else (i.e. positive values) go through unchanged. In the case of the sigmoid, the values are squashed to fit into a limited range between 0 and 1, so this will force the activations to be between 0 and 1 (which are the inputs to the next layer, and so on). Of course, this will lead to more multiplications between numbers in the range $[0, 1]$, which, if multiplied, become even smaller.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-11T15:20:17.520Z
Answers.Text: I think the word "skip" is missleading.  The layers between l and l+t are also processed as normal with weighting and activations. What happens is that the input is added at the layer l+t and then activation on this is done. So, I do not understand that the gradient in backpropagation skips from layer l+t to t (there are other layers in between where the gradient needs to be evaluated)
Answers.UserDisplayName: Herman Jaramillo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-01T14:31:42.820Z
Answers.LastActivityDate: 2021-01-17T14:07:38.757Z
Answers.LastEditDate: 2021-01-17T14:07:38.757Z
Answers.UserDisplayName: nbro
Body: <p>I read that, if we use the sigmoid or hyperbolic tangent activation functions in deep neural networks, we can have some problems with the vanishing of the gradient, and this is visible by the shapes of the derivative of these functions. ReLU solves this problem thanks to its derivative, even if there may be some dead units. ResNet uses ReLU as activation function, but looking online what I understood is that ResNet solves the vanishing of the gradient thanks to its identity map, and I do not totally agree with that. So what's the purpose of the identity connections in ResNet? Are they used for solving the vanishing of the gradient? And ReLU really solves the vanishing of the gradient in deep neural networks?</p>
  
Tags: <deep-learning><activation-functions><exploding-gradient-problem><residual-networks><vanishing-gradient-problem>
Title: Why do ResNets avoid the vanishing gradient problem?
UserDisplayName: FraMan
ThreadText: # Why do ResNets avoid the vanishing gradient problem?
  
  (Asked by: username_0 on 2020-01-30)
  
  I read that, if we use the sigmoid or hyperbolic tangent activation functions in deep neural networks, we can have some problems with the vanishing of the gradient, and this is visible by the shapes of the derivative of these functions. ReLU solves this problem thanks to its derivative, even if there may be some dead units. ResNet uses ReLU as activation function, but looking online what I understood is that ResNet solves the vanishing of the gradient thanks to its identity map, and I do not totally agree with that. So what's the purpose of the identity connections in ResNet? Are they used for solving the vanishing of the gradient? And ReLU really solves the vanishing of the gradient in deep neural networks?
  
  **Tags:** deep-learning, activation-functions, exploding-gradient-problem, residual-networks, vanishing-gradient-problem
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-01. Score: 7 | Accepted answer
  
  *Before proceeding, it's important to note that ResNets, as pointed out [here](https://ai.stackexchange.com/q/18703/2444), were not introduced to *specifically* solve the VGP, but to improve learning in general. In fact, the authors of ResNet, in the original paper, noticed that neural networks without residual connections don't learn as well as ResNets, although they are using batch normalization, which, in theory, ensures that gradients should not vanish ([section 4.1](https://arxiv.org/pdf/1512.03385.pdf#page=5)). So, in this answer, I'm just giving a potential explanation of why ResNets **may also** mitigate (or prevent to some extent) the VGP, but the cited research papers below also confirm that ResNets prevent the VGP. Given that I didn't fully read all the papers mentioned in this answer, the information in this answer may not be fully accurate.*
  
  The skip connections allow information to skip layers, so, in the forward pass, information from layer $l$ can directly be fed into layer $l+t$ (i.e. the activations of layer $l$ are added to the activations of layer $l+t$), for $t \geq 2$, and, during the forward pass, the gradients can also flow unchanged from layer $l+t$ to layer $l$.
  
  How exactly could this prevent the vanishing gradient problem (VGP)? The VGP occurs when the elements of the gradient (the partial derivatives with respect to the parameters of the NN) become exponentially small, so that the update of the parameters with the gradient becomes almost insignificant (i.e. if you add a very small number $0 < \epsilon \ll 1$ to another number $d$, $d+\epsilon$ is almost the same as $d$) and, consequently, the NN learns very slowly or not at all (considering also numerical errors). Given that these partial derivatives are computed with the [chain rule](https://en.wikipedia.org/wiki/Chain_rule), this can easily occur, because you keep on multiplying small (finite-precision) numbers (please, have a look at how the chain rule works, if you're not familiar with it). For example, $\frac{1}{5}\frac{1}{5} = \frac{1}{25}$ and then $\frac{1}{5}\frac{1}{25} = \frac{1}{125}$, and so on. The deeper the NN, the more likely the VGP can occur. This should be quite intuitive if you are familiar with the chain rule and the back-propagation algorithm (i.e. the chain rule). By allowing information to skip layers, layer $l+t$ receives information from both layer $l+t-1$ and layer $l$ (unchanged, i.e. you do not perform multiplications). For example, to compute the activation of layer $l+t-1$, you perform the usual linear combination followed by the application of the non-linear activation function (e.g. ReLU). In this linear combination, you perform multiplications between numbers that could already be quite small, so the results of these multiplications are even smaller numbers. If you use [saturating activation functions (e.g. tanh)](https://stats.stackexchange.com/q/174295/82135), this problem can even be aggravated. If the activation of layer $l+t$ are even smaller than the activations of layer $l+t-1$, the addition of the information from layer $l$ will make these activations bigger, thus, to some extent, they will prevent these activations from becoming exponentially small. A similar thing can be said for the back-propagation of the gradient.
  
  Therefore, skip connections can mitigate the VGP, and so they can be used to train deeper NNs.
  
  These explanations are roughly consistent with the findings reported in the paper [Residual Networks Behave Like Ensembles of Relatively Shallow Networks](https://arxiv.org/pdf/1605.06431.pdf), which states
  
  > Our results reveal one of the key characteristics that seem to enable the training of very deep networks: Residual networks avoid the vanishing gradient problem by introducing **short paths** which can carry gradient throughout the extent of very deep networks.
  
  In the paper [Norm-Preservation: Why Residual Networks Can Become Extremely Deep?](https://arxiv.org/pdf/1805.07477.pdf), the authors also discuss another desirable effect of skip connections.
  
  > We show theoretically and empirically that each residual block in ResNets is increasingly **norm-preserving**, as the network becomes deeper
  
  > **Comments:**
  > **username_0**: Thank you, as I can see, it is not ReLU that do the main job, but the skip connections  
  > **username_1**: @FraMan I would say that both contribute to mitigating the problem. The ReLU, as opposed to e.g. the sigmoid, does not squash its inputs. It basically only sets negative values to zero and lets everything else (i.e. positive values) go through unchanged. In the case of the sigmoid, the values are squashed to fit into a limited range between 0 and 1, so this will force the activations to be between 0 and 1 (which are the inputs to the next layer, and so on). Of course, this will lead to more multiplications between numbers in the range $[0, 1]$, which, if multiplied, become even smaller.  
  > **username_2**: I think the word "skip" is missleading.  The layers between l and l+t are also processed as normal with weighting and activations. What happens is that the input is added at the layer l+t and then activation on this is done. So, I do not understand that the gradient in backpropagation skips from layer l+t to t (there are other layers in between where the gradient needs to be evaluated)  
  
  ---
  
  