Answers.Body: <p>They explained in the paper why they introduce residual blocks. They argue that it's easier to learn residual functions <span class="math-container">$F(x) = H(x) - x$</span> and then add them to the original representation <span class="math-container">$x$</span> to get hidden representation <span class="math-container">$H(x) = F(x) + x$</span> than it is to learn hidden representation <span class="math-container">$H(x)$</span> directly from original representation. That's the main reason and empirical results show that they might be right. Better gradient propagation might be an additional bonus but that's not why they originally introduced the idea.</p>
  <p>&quot;Normal&quot; networks work too but, at some point, they become too deep and start working worse than shallower versions (they empirically showed that in the paper). Again, they argue that the reason for that might be that at deeper layers hidden representations become approximately similar <span class="math-container">$H_n \approx H_{n+1}$</span> because representation is already well learned and you only need some slight adjustments.  That would mean that transformation for deeper layers is similar to identity transformation and that ordinary layers might have trouble learning that, while for residual blocks it would be easy to learn slight modification and add that to the already existing representation from the previous layer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-20T14:19:40.057Z
Answers.Text: Thanks! I'm almost there. It's just that I wonder why the non-residual version gets close to good mapping but doesn't quite get it. I feel like maybe it's because it settles on a minimum in hp space which is not the correct one, and actually quite far from the correct one. Whereas maybe when mapping a residual, you get dropped in at least the same town as the global minimum, so any local minimum you find is just a few blocks away.
  And this explanation doesn't rely on vanishing gradients at all.
Answers.UserDisplayName: Alexander Soare
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-20T17:45:15.593Z
Answers.Text: Well I don't know that's a lot of speculation. You shouldn't take all of this to literally its just an interpretation. The way I looked at it, if you look at the identity operator, that is an identity matrix with ones on diagonal and 0 elsewhere. Now if that matrix represents weights, then diagonal weight should be really large (around 1) and others should be very small (around 0). Those are not very balanced weight relations so NN might have trouble learning that. With residuals you wouldn't have that problem because you're adding to original representation so all weights would be small.
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-06T23:36:39.510Z
Answers.Text: I like the idea of identity propagator! In quantum mechanics the time evolution operator for very short time can be expressed as the sum of an identity operator and the Hamiltonian. So in the very short time intervals after the time evolution starts, the system's evolution can be expressed by an identity operator and Hamiltonian operator.
  In neural networks, I can imagine that as we go from layer to another layer, the representations or weights of layers change gradually which hints the next representation can be obtained by an identity operator and some nonlinear operator.
Answers.UserDisplayName: MOON
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-18T08:53:09.867Z
Answers.LastActivityDate: 2021-01-17T14:11:29.100Z
Answers.LastEditDate: 2021-01-17T14:11:29.100Z
Answers.UserDisplayName: Brale
Body: <p>I often see blog posts or <a href="https://ai.stackexchange.com/questions/17764/why-do-resnets-avoid-the-vanishing-gradient-problem">questions on here</a> starting with the premise that ResNets solve the vanishing gradient problem.</p>
  <p>The <a href="https://arxiv.org/pdf/1512.03385v1.pdf" rel="nofollow noreferrer">original 2015 paper</a> contains the following passage in section 4.1:</p>
  <blockquote>
  <p>We argue that this optimization difficulty is unlikely to
  be caused by vanishing gradients. These plain networks are
  trained with BN, which ensures forward propagated
  signals to have non-zero variances. We also verify that the
  backward propagated gradients exhibit healthy norms with
  BN. So neither forward nor backward signals vanish. In
  fact, the 34-layer plain net is still able to achieve competitive accuracy, suggesting that the solver works
  to some extent.</p>
  </blockquote>
  <p>So what's happened since then? I feel like either it became a misconception that ResNets solve the vanishing gradient problem (because it does indeed feel like a sensible explanation that one would readily accept and continue to propagate), or some paper has since proven that this is indeed the case.</p>
  <p>I'm starting with the initial knowledge that it's &quot;easier&quot; to learn the residual mapping for a convolutional block than it is to learn the whole mapping. So my question is on the level of: why is it &quot;easier&quot;? And why does the &quot;plain network&quot; do such a good job but then struggle to close the gap to the performance of ResNet. Supposedly if the plain network has already learned reasonably good mappings, then all it has left to learn to close the gap is &quot;residual&quot;. But it just isn't able to.</p>
  
Tags: <convolutional-neural-networks><papers><residual-networks><vanishing-gradient-problem>
Title: If vanishing gradients are NOT the problem that ResNets solve, then what is the explanation behind ResNet success?
UserDisplayName: Alexander Soare
ThreadText: # If vanishing gradients are NOT the problem that ResNets solve, then what is the explanation behind ResNet success?
  
  (Asked by: username_0 on 2020-03-18)
  
  I often see blog posts or [questions on here](https://ai.stackexchange.com/questions/17764/why-do-resnets-avoid-the-vanishing-gradient-problem) starting with the premise that ResNets solve the vanishing gradient problem.
  
  The [original 2015 paper](https://arxiv.org/pdf/1512.03385v1.pdf) contains the following passage in section 4.1:
  
  > We argue that this optimization difficulty is unlikely to
  > be caused by vanishing gradients. These plain networks are
  > trained with BN, which ensures forward propagated
  > signals to have non-zero variances. We also verify that the
  > backward propagated gradients exhibit healthy norms with
  > BN. So neither forward nor backward signals vanish. In
  > fact, the 34-layer plain net is still able to achieve competitive accuracy, suggesting that the solver works
  > to some extent.
  
  So what's happened since then? I feel like either it became a misconception that ResNets solve the vanishing gradient problem (because it does indeed feel like a sensible explanation that one would readily accept and continue to propagate), or some paper has since proven that this is indeed the case.
  
  I'm starting with the initial knowledge that it's "easier" to learn the residual mapping for a convolutional block than it is to learn the whole mapping. So my question is on the level of: why is it "easier"? And why does the "plain network" do such a good job but then struggle to close the gap to the performance of ResNet. Supposedly if the plain network has already learned reasonably good mappings, then all it has left to learn to close the gap is "residual". But it just isn't able to.
  
  **Tags:** convolutional-neural-networks, papers, residual-networks, vanishing-gradient-problem
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-03-18. Score: 6 | Accepted answer
  
  They explained in the paper why they introduce residual blocks. They argue that it's easier to learn residual functions $F(x) = H(x) - x$ and then add them to the original representation $x$ to get hidden representation $H(x) = F(x) + x$ than it is to learn hidden representation $H(x)$ directly from original representation. That's the main reason and empirical results show that they might be right. Better gradient propagation might be an additional bonus but that's not why they originally introduced the idea.
  
  "Normal" networks work too but, at some point, they become too deep and start working worse than shallower versions (they empirically showed that in the paper). Again, they argue that the reason for that might be that at deeper layers hidden representations become approximately similar $H\_n \approx H\_{n+1}$ because representation is already well learned and you only need some slight adjustments. That would mean that transformation for deeper layers is similar to identity transformation and that ordinary layers might have trouble learning that, while for residual blocks it would be easy to learn slight modification and add that to the already existing representation from the previous layer.
  
  > **Comments:**
  > **username_0**: Thanks! I'm almost there. It's just that I wonder why the non-residual version gets close to good mapping but doesn't quite get it. I feel like maybe it's because it settles on a minimum in hp space which is not the correct one, and actually quite far from the correct one. Whereas maybe when mapping a residual, you get dropped in at least the same town as the global minimum, so any local minimum you find is just a few blocks away. And this explanation doesn't rely on vanishing gradients at all.  
  > **username_1**: Well I don't know that's a lot of speculation. You shouldn't take all of this to literally its just an interpretation. The way I looked at it, if you look at the identity operator, that is an identity matrix with ones on diagonal and 0 elsewhere. Now if that matrix represents weights, then diagonal weight should be really large (around 1) and others should be very small (around 0). Those are not very balanced weight relations so NN might have trouble learning that. With residuals you wouldn't have that problem because you're adding to original representation so all weights would be small.  
  > **username_2**: I like the idea of identity propagator! In quantum mechanics the time evolution operator for very short time can be expressed as the sum of an identity operator and the Hamiltonian. So in the very short time intervals after the time evolution starts, the system's evolution can be expressed by an identity operator and Hamiltonian operator. In neural networks, I can imagine that as we go from layer to another layer, the representations or weights of layers change gradually which hints the next representation can be obtained by an identity operator and some nonlinear operator.  
  
  ---
  
  