Answers.Body: <p>Your idea is exactly the idea behind state-space models. They have a linear &quot;residual&quot; connection from previous hidden states, skipping activations. In fact, it works very well! I'd recommend reading the <a href="https://arxiv.org/abs/2312.00752" rel="nofollow noreferrer">Mamba paper</a>. They get comparable results to &quot;transformers&quot; while being several times more efficient, and fixing the context-length issue.</p>
  <p>There are two major issues with LSTMs, and in general RNNs, that make training slow:</p>
  <ol>
  <li>Vanishing gradients (as you mentioned), and</li>
  <li>Only being able to train one token in a sequence at once.</li>
  </ol>
  <p>Because state-space models just have a linear &quot;residual&quot; connection, they can train an entire sequence of tokens at once with a parallel scan algorithm (similar to the <a href="https://en.wikipedia.org/wiki/Prefix_sum" rel="nofollow noreferrer">prefix sum algorithm</a>). Note that this ability to crunch huge amounts of data is what allowed &quot;transformers&quot; to take off.</p>
  <p>The issue you mentioned is actually a little broader: if you run an RNN too long the outputs will either vanish <em>or</em> explode. You can imagine a purely linear model, <span class="math-container">$x\mapsto Ax$</span>, which after <span class="math-container">$n$</span> steps will become <span class="math-container">$A^{n}x$</span>. If the eigenvalues are too big, things explode, while if they are too small, <span class="math-container">$A^{n}x\to 0$</span>. The authors of Mamba considered forcing <span class="math-container">$A$</span> to be unitary (eigenvalues have magnitude exactly one, see section B.3), and mention that other authors have explored it in more depth to resolve this issue.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-26T03:59:37.593Z
Answers.LastActivityDate: 2024-03-26T03:59:37.593Z
Answers.UserDisplayName: programjames
Answers.Body: <p>In my opinion your idea indeed holds merit. Something worth noting though is that it is cruder than the LSTM/GRU that have trainable weights that guide what features are remembered and forgotten. There are also other architectures than ResNet that uses residuals, but add gate layers to guide what is passed on.</p>
  <p>The main thing your idea would bring is likely speed, rather than accuracy. And that is not always a bad thing.</p>
  <p>Here are some papers where the author uses a residuals in combination with LSTM:<br>
  <a href="https://arxiv.org/abs/1701.03360" rel="nofollow noreferrer">Residual LSTM</a><br>
  <a href="https://static.googleusercontent.com/media/research.google.com/sv//pubs/archive/46171.pdf" rel="nofollow noreferrer">Highway-LSTM</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-25T18:58:05.137Z
Answers.LastActivityDate: 2024-03-25T18:58:05.137Z
Answers.UserDisplayName: Victor Björkgren
Body: <p>I was recently brushing up on my deep-learning basics and came back to RNNs.  LSTMs/GRUs and the Transformer architecture were invented to solve RNN's vanishing/exploding gradient problem. I was at the time also interested in image processing and the ResNet model, so I immediately thought of incorporating the residual connection into RNNs. <strong>The idea is to connect the hidden states after each block (skipping the network and activations) to allow for better training of the past blocks.</strong> Given that nobody has done this yet, I feel the idea is somehow flawed and not as genius as my mind thinks. Why would/wouldn't this work? Where did I make a mistake?</p>
  
Tags: <recurrent-neural-networks><long-short-term-memory><sequence-modeling><residual-networks><vanishing-gradient-problem>
Title: Wouldn't residual connections in RNNs solve the vanishing/exploding gradient problem?
UserDisplayName: Vladislav Korecký
ThreadText: # Wouldn't residual connections in RNNs solve the vanishing/exploding gradient problem?
  
  (Asked by: username_0 on 2024-03-25)
  
  I was recently brushing up on my deep-learning basics and came back to RNNs. LSTMs/GRUs and the Transformer architecture were invented to solve RNN's vanishing/exploding gradient problem. I was at the time also interested in image processing and the ResNet model, so I immediately thought of incorporating the residual connection into RNNs. **The idea is to connect the hidden states after each block (skipping the network and activations) to allow for better training of the past blocks.** Given that nobody has done this yet, I feel the idea is somehow flawed and not as genius as my mind thinks. Why would/wouldn't this work? Where did I make a mistake?
  
  **Tags:** recurrent-neural-networks, long-short-term-memory, sequence-modeling, residual-networks, vanishing-gradient-problem
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-03-25. Score: 3 | Accepted answer
  
  In my opinion your idea indeed holds merit. Something worth noting though is that it is cruder than the LSTM/GRU that have trainable weights that guide what features are remembered and forgotten. There are also other architectures than ResNet that uses residuals, but add gate layers to guide what is passed on.
  
  The main thing your idea would bring is likely speed, rather than accuracy. And that is not always a bad thing.
  
  Here are some papers where the author uses a residuals in combination with LSTM:  
  [Residual LSTM](https://arxiv.org/abs/1701.03360)  
  [Highway-LSTM](https://static.googleusercontent.com/media/research.google.com/sv//pubs/archive/46171.pdf)
  
  
  ---
  
  ### Answer by username_1 on 2024-03-26. Score: 2
  
  Your idea is exactly the idea behind state-space models. They have a linear "residual" connection from previous hidden states, skipping activations. In fact, it works very well! I'd recommend reading the [Mamba paper](https://arxiv.org/abs/2312.00752). They get comparable results to "transformers" while being several times more efficient, and fixing the context-length issue.
  
  There are two major issues with LSTMs, and in general RNNs, that make training slow:
  
  1. Vanishing gradients (as you mentioned), and
  2. Only being able to train one token in a sequence at once.
  
  Because state-space models just have a linear "residual" connection, they can train an entire sequence of tokens at once with a parallel scan algorithm (similar to the [prefix sum algorithm](https://en.wikipedia.org/wiki/Prefix_sum)). Note that this ability to crunch huge amounts of data is what allowed "transformers" to take off.
  
  The issue you mentioned is actually a little broader: if you run an RNN too long the outputs will either vanish *or* explode. You can imagine a purely linear model, $x\mapsto Ax$, which after $n$ steps will become $A^{n}x$. If the eigenvalues are too big, things explode, while if they are too small, $A^{n}x\to 0$. The authors of Mamba considered forcing $A$ to be unitary (eigenvalues have magnitude exactly one, see section B.3), and mention that other authors have explored it in more depth to resolve this issue.
  
  
  ---
  
  