Answers.Body: <p>RNN's/LSTM's work better on smaller datasets compared to transformers, assuming neither are pretrained.</p>
  <p>RNN's perform worse than transformers in most tasks with enough data due to RNN's having a higher model bias.
  Recall that in many ML classes, you are taught that with enough samples, models with higher bias (and lower complexity) will perform worse than models with lower bias (and higher complexity).
  RNN's take in a sequence of elements <span class="math-container">$x_1,x_2,...,x_n$</span> and assume a markov property. this markov property is used to simplify / reduce the complexity of representations for the input. That and the fact that the RNN only uses one representation for the sequence of tokens are examples of simplifying/introducing a bias into the model architecture.
  Transformers create a representation for each token, and do not assume that markov property.
  This means it has lower bias than RNN's and can scale better with more data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-18T03:49:44.833Z
Answers.LastActivityDate: 2023-05-18T03:51:58.767Z
Answers.LastEditDate: 2023-05-18T03:51:58.767Z
Answers.UserDisplayName: jeb
Answers.Body: <p>LSTMs were the state-of-the-art (SOTA) in many cases (e.g. machine translation) until transformers came along - now I don't really know the SOTA or where LSTMs still perform better than e.g. transformers. LSTMs were introduced to solve the <strong>vanishing</strong> and <strong>exploding gradient problems</strong>. Even <a href="http://www.bioinf.jku.at/publications/older/2604.pdf" rel="nofollow noreferrer">the LSTM paper</a> tells you that</p>
  <blockquote>
  <p>In comparisons with RTRL, BPTT, Recurrent Cascade-Correlation,
  Elman nets, and Neural Sequence Chunking, LSTM leads to many <strong>more successful runs, and learns much faster. LSTM also solves complex, articial long time lag tasks that have never been solved by previous recurrent network algorithms</strong>.</p>
  </blockquote>
  <p>For a specific case where LSTM achieved SOTA (if I remember correctly), you can check <a href="https://arxiv.org/pdf/1409.0473.pdf" rel="nofollow noreferrer">the <strong>neural machine translation</strong> paper</a>. Google used LSTMs for some time in Google Translate. See <a href="https://research.google/pubs/pub45610/" rel="nofollow noreferrer">this paper</a> for more details.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-27T17:16:32.790Z
Answers.Text: The original LSTM paper is quite old, which is why I was a bit skeptical about their conclusions holding up against more modern architectures (transformers aside). Maybe things haven't changed as much as I thought? From what I can tell, the neural machine translation paper doesn't use LSTMs . I appreciate the Google Translate reference, quite interesting! I was hoping for a more head-to-head comparison, which the Google translate paper appears to do. Thank you!
Answers.UserDisplayName: Vladimir Belik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-27T17:37:55.210Z
Answers.Text: GRUs can be viewed as a variation of LSTMs. See also [this repo](https://github.com/tensorflow/nmt).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-26T21:19:37.977Z
Answers.LastActivityDate: 2022-06-26T21:19:37.977Z
Answers.UserDisplayName: nbro
Body: <p>I have heard a lot of hype around LSTM for all kinds of time-series based applications including NLP. Despite this, I haven't seen many (if any) applications of LSTM where LSTM performs <em><strong>uniquely</strong></em> well compared to other type of deep learning, including more vanilla RNN.</p>
  <p>Are there any examples where LSTM does significantly better on a particular task, compared to other modern algorithms and architectures?</p>
  
Tags: <deep-learning><reference-request><long-short-term-memory><applications>
Title: Does LSTM provide any unique value or advantages compared to other algorithms, including "vanilla" RNN?
UserDisplayName: Vladimir Belik
ThreadText: # Does LSTM provide any unique value or advantages compared to other algorithms, including "vanilla" RNN?
  
  (Asked by: username_0 on 2022-06-26)
  
  I have heard a lot of hype around LSTM for all kinds of time-series based applications including NLP. Despite this, I haven't seen many (if any) applications of LSTM where LSTM performs ***uniquely*** well compared to other type of deep learning, including more vanilla RNN.
  
  Are there any examples where LSTM does significantly better on a particular task, compared to other modern algorithms and architectures?
  
  **Tags:** deep-learning, reference-request, long-short-term-memory, applications
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-06-26. Score: 3 | Accepted answer
  
  LSTMs were the state-of-the-art (SOTA) in many cases (e.g. machine translation) until transformers came along - now I don't really know the SOTA or where LSTMs still perform better than e.g. transformers. LSTMs were introduced to solve the **vanishing** and **exploding gradient problems**. Even [the LSTM paper](http://www.bioinf.jku.at/publications/older/2604.pdf) tells you that
  
  > In comparisons with RTRL, BPTT, Recurrent Cascade-Correlation,
  > Elman nets, and Neural Sequence Chunking, LSTM leads to many **more successful runs, and learns much faster. LSTM also solves complex, articial long time lag tasks that have never been solved by previous recurrent network algorithms**.
  
  For a specific case where LSTM achieved SOTA (if I remember correctly), you can check [the **neural machine translation** paper](https://arxiv.org/pdf/1409.0473.pdf). Google used LSTMs for some time in Google Translate. See [this paper](https://research.google/pubs/pub45610/) for more details.
  
  > **Comments:**
  > **username_0**: The original LSTM paper is quite old, which is why I was a bit skeptical about their conclusions holding up against more modern architectures (transformers aside). Maybe things haven't changed as much as I thought? From what I can tell, the neural machine translation paper doesn't use LSTMs . I appreciate the Google Translate reference, quite interesting! I was hoping for a more head-to-head comparison, which the Google translate paper appears to do. Thank you!  
  > **username_2**: GRUs can be viewed as a variation of LSTMs. See also [this repo](https://github.com/tensorflow/nmt).  
  
  ---
  
  ### Answer by username_1 on 2023-05-18. Score: 1
  
  RNN's/LSTM's work better on smaller datasets compared to transformers, assuming neither are pretrained.
  
  RNN's perform worse than transformers in most tasks with enough data due to RNN's having a higher model bias.
  Recall that in many ML classes, you are taught that with enough samples, models with higher bias (and lower complexity) will perform worse than models with lower bias (and higher complexity).
  RNN's take in a sequence of elements $x\_1,x\_2,...,x\_n$ and assume a markov property. this markov property is used to simplify / reduce the complexity of representations for the input. That and the fact that the RNN only uses one representation for the sequence of tokens are examples of simplifying/introducing a bias into the model architecture.
  Transformers create a representation for each token, and do not assume that markov property.
  This means it has lower bias than RNN's and can scale better with more data.
  
  
  ---
  
  