Answers.Body: <p>First a little intro, skip to the end for the straight answers: residual networks were proposed after observing that deeper models tend to perform worse than their shallow counterpart if we just keep adding hidden layers without applying any other change to the architecture, as we can see in the very first picture of the original <a href="https://arxiv.org/pdf/1512.03385.pdf" rel="nofollow noreferrer">paper</a>.
  <a href="https://i.sstatic.net/Yl6rK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Yl6rK.png" alt="enter image description here" /></a></p>
  <p>The reason of this phenomena is indeed gradient vanishing. The more the hidden layers, the more the information of the original input get lost, due to the fact that a hidden layer receives only information from the previous hidden layer. How to solve this? Using residual connections. A residual connection is just an identity function that map an input or hidden state forward in the network, so not to the immediate next layers, that's why these connections are also called skip connections. The only purpose they serve is to force deep layers to retain information learned in the early layers of the network.</p>
  <p>From a numerical perspective you can think about information getting lost as weights becoming smaller and smaller. By brutally summing the hidden states of previous layers you make sure to avoid this problem, giving the weights a broader range of adjustment even in very deep layers.</p>
  <p>In conclusion, to answer your questions:</p>
  <p><strong>1</strong> Residual connection don't create or learn an identity function, they simply use it. The formulation of such connections in the paper is:</p>
  <p><span class="math-container">$y = F(x, W_{i}) + x$</span></p>
  <p>where x could be rewritten as <span class="math-container">$I(x)$</span>, <span class="math-container">$I$</span> being the identity function.</p>
  <p><strong>2</strong> No, we don't loose any information by summing the residuals, on the contrary, they are designed to retaining information also in very deep layers, for the above mentioned reasons.</p>
  <p><strong>3</strong> All layers are updated, there's no frozen layers in a residual network. The &quot;skip&quot; term refers to the fact that a hidden layers is copied in forward layers, which is a legit operation, but it doesn't refer at all to skip in training or weight updates.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-30T09:00:46.197Z
Answers.Text: Thanks for the detailed description. I have a quick question. I'm reading that the goal of the residual block so that before the last 'ReLU' activation, the summation of the input and the output of the residual block, should equal. So that is why I am confused that to make this happen you need the residual block to output 0. Also, does summating the input with the output of the residual block not alter the true learning of the residual block? or is the purpose to just mix in information previously learnt with information learnt in the residual block. Thanks.
Answers.UserDisplayName: user49443
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-31T12:20:50.747Z
Answers.Text: "the goal of the residual block so that before the last 'ReLU' activation, the summation of the input and the output of the residual block, should equal ". The summation is not suppose to be zero, the 'skip connection' before the relu actually prevent everything going to zero, due to vanishing gradient.  "is the purpose to just mix in information previously learnt with information learnt in the residual block" Yes, precisely. Thanks to the skip connections a deep layer would at the very least output the same as previous layers, but never loose information.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-27T13:32:03.047Z
Answers.LastActivityDate: 2021-08-27T20:15:52.560Z
Answers.LastEditDate: 2021-08-27T20:15:52.560Z
Answers.UserDisplayName: Edoardo Guerriero
Answers.Body: <p>The residual layer was not invented to solve the vanishing gradient problem. Citing from the official <a href="https://arxiv.org/pdf/1512.03385.pdf" rel="nofollow noreferrer">ResNet paper</a>:</p>
  <blockquote>
  <p>An obstacle to answering this question was the notorious problem of vanishing/exploding gradients [1, 9], which hamper convergence from the beginning. This problem, however, has been largely addressed by normalized initialization [23, 9, 37, 13] and intermediate normalization layers [16], which enable networks with tens of layers to start converging for stochastic gradient descent (SGD) with backpropagation [22].</p>
  </blockquote>
  <p>The problem that this layer solves is: <em>How to create deeper models so that they perform definitely better than shallower models?</em></p>
  <p>By adding the residual connection we modify the output of the layer so that instead of <span class="math-container">$F(x) = f(x)$</span> it now outputs <span class="math-container">$F(x) = f(x) + x$</span>, where <span class="math-container">$f$</span> is the function represented by the layer around which we add the skip-connection.</p>
  <p><a href="https://i.sstatic.net/3Gb8z.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3Gb8z.png" alt="residual connection" /></a></p>
  <p><em>(technically the skip-connection is considered the main flow, and the NN layer is called the residual)</em></p>
  <p>Now if you have two identical models, but one has <span class="math-container">$N$</span> layers and the other <span class="math-container">$N+1$</span> layers, and <em><strong>if the shallower model was the better choice</strong></em>, then the last layer of the deeper model is actually not needed (or one of the layers, no need to be the last). In order for the deeper model to simulate the shallower model it simply has to learn that the function of the last layer has to be <span class="math-container">$f(x) = 0$</span>, i.e. all of the weights of the last layer have to be <span class="math-container">$0$</span>. <strong>Without the skip-connection</strong> the model has to learn that the last layer function has to be <span class="math-container">$f(x) = x$</span>, which is much more difficult.</p>
  <p>So now if your model is not performing well you can simply add more layers and you expect that the performance will not degrade. It may not improve, but it should not degrade.</p>
  <p>See this if you want to read more: <a href="https://pi-tau.github.io/posts/res-nets/" rel="nofollow noreferrer">https://pi-tau.github.io/posts/res-nets/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-24T13:14:39.803Z
Answers.LastActivityDate: 2023-07-27T11:26:53.413Z
Answers.LastEditDate: 2023-07-27T11:26:53.413Z
Answers.UserDisplayName: pi-tau
Answers.Body: <p>For the Third question,</p>
  <blockquote>
  <p>I believe it also helps with backpropagation so that it doesn't have to update all the weights layer by layer and it can skip back to shallow layers. Therefore, are the weights inside the residual block, i.e layers 11-15 not updated? If not, then what is the point of the 11-15th layer if they are not designed to &quot;do anything&quot;.</p>
  </blockquote>
  <p>Model with many layers, it is easy to get a phenomenon- gradient vanishing. In short, model does not learning in some layers(as layers 11-15 in your example). And Residual Block can help to solve this phenomenon.</p>
  <p>So, we can clarify it. These layers are designed and we hope they could work well. But in some situation, not well --&gt; adding Residual Block is a solution.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-20T03:46:29.353Z
Answers.LastActivityDate: 2022-04-20T03:46:29.353Z
Answers.UserDisplayName: Tan Phan
Body: <p>I've learnt that idea that the residual block was invented to solve the vanishing gradient problem due to the deep layer to layer multiplication.</p>
  <p>I understand that for example if I have 10 layers, and I add another 5 layers, that the output of the 10th layer will 'skip' the 5 layers. Although, the output of the 10th layer will also pass through the 5 layers as well. Just before the 15th layer Relu, the output from the 10th layer is element-wise summed with the 15th layer, just prior to the final Relu.
  I have some confusion with this.</p>
  <ol>
  <li><p>Identity mapping/function. I keep reading that it creates an identiy function or it learns an identity function. What exactly is this? Isn't is just F(x) = 5 added layers, and x =output of 10th layer and thus it is just F(X) + X?</p>
  </li>
  <li><p>By summing the output of the 10th layer to the 15th layer, will this not affect what was learnt in the 5 layers? I.e. from 11th -15th layer.</p>
  </li>
  <li><p>I believe it also helps with backpropagation so that it doesn't have to update all the weights layer by layer and it can skip back to shallow layers. Therefore, are the weights inside the residual block, i.e layers 11-15 not updated? If not, then what is the point of the 11-15th layer if they are not designed to &quot;do anything&quot;.</p>
  </li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-26T15:17:05.683Z
Comments.Text: An identity function is f(x)=x so F(X)+X is the same as F(X)+f(X) if f is the identity function
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-27T05:18:55.447Z
Comments.Text: @user253751 Yes I know that, but why do we train the residual block to output 0? Because we need it to be 0 such that x the identity, can be the same as the input to the residual block.
Comments.UserDisplayName: user49443
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-27T17:20:34.350Z
Comments.Text: The purpose of a residual block is not to output zero, I am not sure where you got that from.
Comments.UserDisplayName: Dr. Snoopy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-30T09:02:50.567Z
Comments.Text: @Dr.Snoopy I keep reading that the output should equal to input. And the equation ``F(x) = f(x) + x`` keeps getting brought up and they say the point of identity is the input should be the same as the output, so in my head that means the residual block needs to be 0, or learn to make it 0. because ``F(x) = 0 + x`` which is ``F(x) = x``
Comments.UserDisplayName: user49443
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-30T11:32:11.273Z
Comments.Text: Keep reading, where? That is not the correct interpretation, I think you are reading about a different kind of residual.
Comments.UserDisplayName: Dr. Snoopy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-30T23:03:50.987Z
Comments.Text: @Dr.Snoopy https://towardsdatascience.com/resnets-residual-blocks-deep-residual-learning-a231a0ee73d2 
  
  "In the above equation in figure 3, let g be the ReLU activation function. If the activations for the layer l+2 tends to 0"  - This is what confuses me
Comments.UserDisplayName: user49443
Tags: <neural-networks><machine-learning><residual-networks>
Title: Residual Blocks - why do they work?
UserDisplayName: user49443
ThreadText: # Residual Blocks - why do they work?
  
  (Asked by: username_0 on 2021-08-26)
  
  I've learnt that idea that the residual block was invented to solve the vanishing gradient problem due to the deep layer to layer multiplication.
  
  I understand that for example if I have 10 layers, and I add another 5 layers, that the output of the 10th layer will 'skip' the 5 layers. Although, the output of the 10th layer will also pass through the 5 layers as well. Just before the 15th layer Relu, the output from the 10th layer is element-wise summed with the 15th layer, just prior to the final Relu.
  I have some confusion with this.
  
  1. Identity mapping/function. I keep reading that it creates an identiy function or it learns an identity function. What exactly is this? Isn't is just F(x) = 5 added layers, and x =output of 10th layer and thus it is just F(X) + X?
  2. By summing the output of the 10th layer to the 15th layer, will this not affect what was learnt in the 5 layers? I.e. from 11th -15th layer.
  3. I believe it also helps with backpropagation so that it doesn't have to update all the weights layer by layer and it can skip back to shallow layers. Therefore, are the weights inside the residual block, i.e layers 11-15 not updated? If not, then what is the point of the 11-15th layer if they are not designed to "do anything".
  
  **Tags:** neural-networks, machine-learning, residual-networks
  
  > **Comments:**
  > **username_1**: An identity function is f(x)=x so F(X)+X is the same as F(X)+f(X) if f is the identity function  
  > **username_0**: @user253751 Yes I know that, but why do we train the residual block to output 0? Because we need it to be 0 such that x the identity, can be the same as the input to the residual block.  
  > **username_2**: The purpose of a residual block is not to output zero, I am not sure where you got that from.  
  > **username_0**: @username_2 I keep reading that the output should equal to input. And the equation ``F(x) = f(x) + x`` keeps getting brought up and they say the point of identity is the input should be the same as the output, so in my head that means the residual block needs to be 0, or learn to make it 0. because ``F(x) = 0 + x`` which is ``F(x) = x``  
  > **username_2**: Keep reading, where? That is not the correct interpretation, I think you are reading about a different kind of residual.  
  > **username_0**: @username_2 https://towardsdatascience.com/resnets-residual-blocks-deep-residual-learning-a231a0ee73d2   "In the above equation in figure 3, let g be the ReLU activation function. If the activations for the layer l+2 tends to 0"  - This is what confuses me  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2021-08-27. Score: 2
  
  First a little intro, skip to the end for the straight answers: residual networks were proposed after observing that deeper models tend to perform worse than their shallow counterpart if we just keep adding hidden layers without applying any other change to the architecture, as we can see in the very first picture of the original [paper](https://arxiv.org/pdf/1512.03385.pdf).
  [![enter image description here](https://i.sstatic.net/Yl6rK.png)](https://i.sstatic.net/Yl6rK.png)
  
  The reason of this phenomena is indeed gradient vanishing. The more the hidden layers, the more the information of the original input get lost, due to the fact that a hidden layer receives only information from the previous hidden layer. How to solve this? Using residual connections. A residual connection is just an identity function that map an input or hidden state forward in the network, so not to the immediate next layers, that's why these connections are also called skip connections. The only purpose they serve is to force deep layers to retain information learned in the early layers of the network.
  
  From a numerical perspective you can think about information getting lost as weights becoming smaller and smaller. By brutally summing the hidden states of previous layers you make sure to avoid this problem, giving the weights a broader range of adjustment even in very deep layers.
  
  In conclusion, to answer your questions:
  
  **1** Residual connection don't create or learn an identity function, they simply use it. The formulation of such connections in the paper is:
  
  $y = F(x, W\_{i}) + x$
  
  where x could be rewritten as $I(x)$, $I$ being the identity function.
  
  **2** No, we don't loose any information by summing the residuals, on the contrary, they are designed to retaining information also in very deep layers, for the above mentioned reasons.
  
  **3** All layers are updated, there's no frozen layers in a residual network. The "skip" term refers to the fact that a hidden layers is copied in forward layers, which is a legit operation, but it doesn't refer at all to skip in training or weight updates.
  
  > **Comments:**
  > **username_0**: Thanks for the detailed description. I have a quick question. I'm reading that the goal of the residual block so that before the last 'ReLU' activation, the summation of the input and the output of the residual block, should equal. So that is why I am confused that to make this happen you need the residual block to output 0. Also, does summating the input with the output of the residual block not alter the true learning of the residual block? or is the purpose to just mix in information previously learnt with information learnt in the residual block. Thanks.  
  > **username_3**: "the goal of the residual block so that before the last 'ReLU' activation, the summation of the input and the output of the residual block, should equal ". The summation is not suppose to be zero, the 'skip connection' before the relu actually prevent everything going to zero, due to vanishing gradient.  "is the purpose to just mix in information previously learnt with information learnt in the residual block" Yes, precisely. Thanks to the skip connections a deep layer would at the very least output the same as previous layers, but never loose information.  
  
  ---
  
  ### Answer by username_4 on 2023-07-24. Score: 1
  
  The residual layer was not invented to solve the vanishing gradient problem. Citing from the official [ResNet paper](https://arxiv.org/pdf/1512.03385.pdf):
  
  > An obstacle to answering this question was the notorious problem of vanishing/exploding gradients [1, 9], which hamper convergence from the beginning. This problem, however, has been largely addressed by normalized initialization [23, 9, 37, 13] and intermediate normalization layers [16], which enable networks with tens of layers to start converging for stochastic gradient descent (SGD) with backpropagation [22].
  
  The problem that this layer solves is: *How to create deeper models so that they perform definitely better than shallower models?*
  
  By adding the residual connection we modify the output of the layer so that instead of $F(x) = f(x)$ it now outputs $F(x) = f(x) + x$, where $f$ is the function represented by the layer around which we add the skip-connection.
  
  [![residual connection](https://i.sstatic.net/3Gb8z.png)](https://i.sstatic.net/3Gb8z.png)
  
  *(technically the skip-connection is considered the main flow, and the NN layer is called the residual)*
  
  Now if you have two identical models, but one has $N$ layers and the other $N+1$ layers, and ***if the shallower model was the better choice***, then the last layer of the deeper model is actually not needed (or one of the layers, no need to be the last). In order for the deeper model to simulate the shallower model it simply has to learn that the function of the last layer has to be $f(x) = 0$, i.e. all of the weights of the last layer have to be $0$. **Without the skip-connection** the model has to learn that the last layer function has to be $f(x) = x$, which is much more difficult.
  
  So now if your model is not performing well you can simply add more layers and you expect that the performance will not degrade. It may not improve, but it should not degrade.
  
  See this if you want to read more: <https://username_4.github.io/posts/res-nets/>
  
  
  ---
  
  ### Answer by username_5 on 2022-04-20. Score: 0
  
  For the Third question,
  
  > I believe it also helps with backpropagation so that it doesn't have to update all the weights layer by layer and it can skip back to shallow layers. Therefore, are the weights inside the residual block, i.e layers 11-15 not updated? If not, then what is the point of the 11-15th layer if they are not designed to "do anything".
  
  Model with many layers, it is easy to get a phenomenon- gradient vanishing. In short, model does not learning in some layers(as layers 11-15 in your example). And Residual Block can help to solve this phenomenon.
  
  So, we can clarify it. These layers are designed and we hope they could work well. But in some situation, not well --> adding Residual Block is a solution.
  
  
  ---
  
  