Answers.Body: <p>Vanishing gradient problem is indeed present in MLP and CNNs. Please have a look at <a href="https://arxiv.org/pdf/1512.03385.pdf" rel="noreferrer">ResNet</a> paper: before the introduction of residual blocks, the vanishing gradient problem was one of the main limitations to the depth of the networks.
  In RNNs this problem is present also with fewer layers because each layer needs to perform backpropagation through time, so learning is additionally constrained by the sequence length.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-09T16:53:32.927Z
Answers.Text: Thanks for your answer. What I don't really understand is why are RNN especially prone for the vanishing gradient problem? So why is the backpropagation through time as you wrote, especially prone to that?
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-09T16:47:19.143Z
Answers.LastActivityDate: 2024-01-09T16:47:19.143Z
Answers.UserDisplayName: Ciodar
Answers.Body: <p>Two problems for RNNs</p>
  <ol>
  <li>They're typically &quot;deeper&quot; than a MLP since you can unroll a RNN into a stack of layers for each timestep, which exacerbates the problem</li>
  <li>The activation function of choice typically is tanh which has diminishing gradients at the extremes</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-10T08:29:22.330Z
Answers.Text: Thanks mf908 for your answer. Let's say I have a deep MLP and I also use tanh as the activiation function. Would it be as prone to the vanishing gradient problem as a RNN?
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-10T13:37:51.800Z
Answers.Text: @PeterBe The deeper the network, the worse it's going to be. As far as the vanishing gradient issue is concerned, an RNN of k layers over n timesteps is equivalent to a MLP of kn layers. Can you see that this number kn (k times n) can easily be very large? Of course it's theoretically possible to come up with a very deep MLP and a very shallow RNN over very few timesteps, and then you can say "the vanishing gradient issue is worse for this MLP than for this RNN!!", but in general it's much worse for RNNs than for MLP.
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-10T21:54:01.903Z
Answers.Text: Theoretically yes, it's a part of the reason why ReLu is the activation function of choice nowadays, previously it was sigmoid which similarly has diminishing gradient issues
Answers.UserDisplayName: mf908
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-10T01:01:44.500Z
Answers.LastActivityDate: 2024-01-10T01:01:44.500Z
Answers.UserDisplayName: mf908
Answers.Body: <p>No, ResNet were not introduced to solve vanishing gradients, citing from the paper:</p>
  <blockquote>
  <p>An obstacle to answering this question was the notorious
  problem of vanishing/exploding gradients [1, 9], which
  hamper convergence from the beginning. This problem,
  however, has been largely addressed by normalized initialization [23, 9, 37, 13] and intermediate normalization layers
  [16], which enable networks with tens of layers to start converging for stochastic gradient descent (SGD) with backpropagation [22].</p>
  </blockquote>
  <p>However, vanishing gradient happens also for MLP for the same reasons why they happen in RNNs as you can see an unrolled RNN as a MLP at the end of the day: because you stack multiple layer, and if many of them saturate, the gradient will tend to zero</p>
  <p>You can see it from an unrolled RNN:
  <a href="https://i.sstatic.net/kxe0e.png" rel="noreferrer"><img src="https://i.sstatic.net/kxe0e.png" alt="enter image description here" /></a></p>
  <p>Here, the gradient of <span class="math-container">$E_4$</span> with respect to <span class="math-container">$x_0$</span> will have to travel 6 matrix multiplications/non linearities, even though the net is just 1 layer deep.</p>
  <p>If the spectral norm of such matrices is less than one (ie the highest eigenvalue is &lt; 1), then they are contractive, and thus a vector multiplied many times by such matrix will be attracted to 0</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-10T08:28:08.680Z
Answers.Text: Thanks Alberto for your answer. What I still don't understand is why are RNN especially prone for the vanishing gradient problem? I mean if you use a deep MLP it will look similar as in your figure and why is a deep MLP not prone to the vanishing gradient problem as strong as a RNN?
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-10T10:18:08.223Z
Answers.Text: @PeterBe you should be able to see it from the picture. the problem is that the "connection" from the first input (x0) and the last output (En) is as long as the length of the input size... thus even a network with 1 recurrent layer, given a string of length 50, the last output is 50 layer transformation away from the first input (thus is like having a 50 layers deep net)
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-10T15:04:15.220Z
Answers.Text: @PeterBe you're welcome, good luck with your journey
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-09T19:13:58.273Z
Answers.LastActivityDate: 2024-01-09T19:13:58.273Z
Answers.UserDisplayName: Alberto
Body: <p>I would like to know why the vanishing gradient problem especially relevant for a RNN and not a MLP (multi-layer-pereptron). In a MLP you also backpropagate errors and multiple different weigths. If the weights are small, the resulting update in the last layers in the backpropagation will be very small</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-09T16:57:34.213Z
Comments.Text: In the most simple case, you can think of RNN as a recursive MLP that calls itself multiple times. We can unfold it into a very deep MLP and as a result, the gradients vanish as they propagate through the numerous layers.
Comments.UserDisplayName: Aray Karjauv
Tags: <recurrent-neural-networks><vanishing-gradient-problem>
Title: Why is the vanishing gradient problem especially relevant for a RNN and not a MLP
UserDisplayName: PeterBe
ThreadText: # Why is the vanishing gradient problem especially relevant for a RNN and not a MLP
  
  (Asked by: username_0 on 2024-01-09)
  
  I would like to know why the vanishing gradient problem especially relevant for a RNN and not a MLP (multi-layer-pereptron). In a MLP you also backpropagate errors and multiple different weigths. If the weights are small, the resulting update in the last layers in the backpropagation will be very small
  
  **Tags:** recurrent-neural-networks, vanishing-gradient-problem
  
  > **Comments:**
  > **username_1**: In the most simple case, you can think of RNN as a recursive MLP that calls itself multiple times. We can unfold it into a very deep MLP and as a result, the gradients vanish as they propagate through the numerous layers.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_5 on 2024-01-09. Score: 6 | Accepted answer
  
  No, ResNet were not introduced to solve vanishing gradients, citing from the paper:
  
  > An obstacle to answering this question was the notorious
  > problem of vanishing/exploding gradients [1, 9], which
  > hamper convergence from the beginning. This problem,
  > however, has been largely addressed by normalized initialization [23, 9, 37, 13] and intermediate normalization layers
  > [16], which enable networks with tens of layers to start converging for stochastic gradient descent (SGD) with backpropagation [22].
  
  However, vanishing gradient happens also for MLP for the same reasons why they happen in RNNs as you can see an unrolled RNN as a MLP at the end of the day: because you stack multiple layer, and if many of them saturate, the gradient will tend to zero
  
  You can see it from an unrolled RNN:
  [![enter image description here](https://i.sstatic.net/kxe0e.png)](https://i.sstatic.net/kxe0e.png)
  
  Here, the gradient of $E\_4$ with respect to $x\_0$ will have to travel 6 matrix multiplications/non linearities, even though the net is just 1 layer deep.
  
  If the spectral norm of such matrices is less than one (ie the highest eigenvalue is < 1), then they are contractive, and thus a vector multiplied many times by such matrix will be attracted to 0
  
  > **Comments:**
  > **username_0**: Thanks username_5 for your answer. What I still don't understand is why are RNN especially prone for the vanishing gradient problem? I mean if you use a deep MLP it will look similar as in your figure and why is a deep MLP not prone to the vanishing gradient problem as strong as a RNN?  
  > **username_5**: @username_0 you should be able to see it from the picture. the problem is that the "connection" from the first input (x0) and the last output (En) is as long as the length of the input size... thus even a network with 1 recurrent layer, given a string of length 50, the last output is 50 layer transformation away from the first input (thus is like having a 50 layers deep net)  
  > **username_5**: @username_0 you're welcome, good luck with your journey  
  
  ---
  
  ### Answer by username_2 on 2024-01-09. Score: 5
  
  Vanishing gradient problem is indeed present in MLP and CNNs. Please have a look at [ResNet](https://arxiv.org/pdf/1512.03385.pdf) paper: before the introduction of residual blocks, the vanishing gradient problem was one of the main limitations to the depth of the networks.
  In RNNs this problem is present also with fewer layers because each layer needs to perform backpropagation through time, so learning is additionally constrained by the sequence length.
  
  > **Comments:**
  > **username_0**: Thanks for your answer. What I don't really understand is why are RNN especially prone for the vanishing gradient problem? So why is the backpropagation through time as you wrote, especially prone to that?  
  
  ---
  
  ### Answer by username_3 on 2024-01-10. Score: 2
  
  Two problems for RNNs
  
  1. They're typically "deeper" than a MLP since you can unroll a RNN into a stack of layers for each timestep, which exacerbates the problem
  2. The activation function of choice typically is tanh which has diminishing gradients at the extremes
  
  > **Comments:**
  > **username_0**: Thanks username_3 for your answer. Let's say I have a deep MLP and I also use tanh as the activiation function. Would it be as prone to the vanishing gradient problem as a RNN?  
  > **username_4**: @username_0 The deeper the network, the worse it's going to be. As far as the vanishing gradient issue is concerned, an RNN of k layers over n timesteps is equivalent to a MLP of kn layers. Can you see that this number kn (k times n) can easily be very large? Of course it's theoretically possible to come up with a very deep MLP and a very shallow RNN over very few timesteps, and then you can say "the vanishing gradient issue is worse for this MLP than for this RNN!!", but in general it's much worse for RNNs than for MLP.  
  > **username_3**: Theoretically yes, it's a part of the reason why ReLu is the activation function of choice nowadays, previously it was sigmoid which similarly has diminishing gradient issues  
  
  ---
  
  