Answers.Body: <p>The Wikipedia article is more technically correct, in that the term RNN is formally taken to mean &quot;a neural network with recurrent connections&quot;, and that includes many architectures that match this description, including LSTMs.</p>
  <p>However, it is also common to see &quot;RNN&quot; used as a short-hand for a kind of &quot;Vanilla RNN&quot; or &quot;basic RNN&quot;, where one or more layers have weights connecting the layer to itself (its own activations from <span class="math-container">$t-1$</span> are concatenated to the external inputs at <span class="math-container">$t$</span>), and there are no other gates or special combinations, just those recurrent connections.</p>
  <p>Oddly, this basic layer-based RNN archtecture is not listed in all the options on the <a href="https://en.wikipedia.org/wiki/Recurrent_neural_network" rel="noreferrer">Wikipedia page on RNNs</a> - probably the closest are Elman networks and Jordan networks which are ways to implement the recurrent connection. It is a valid architecture choice, and can be effective. The LSTM and GRU architectures improve on it in terms of handling longer sequences and preserving important signals over them when training (e.g. matching a starting and ending quote in text processing).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-13T16:19:01.057Z
Answers.Text: So, I would clarify that this is a matter of terminology. If you define RNN (a short for recurrent neural networks) as the set of all neural networks with recurrent connections, then LSTMs could be considered a subset of RNN. However, if you define RNN as the vanilla RNN, then LSTMs are not a subset of RNNs (because vanilla RNNs do not have gates): it probably would be more the other way around (it may be possible to represent vanilla RNNs as LSTMs, though I don't remember the details now, so I am not really sure if this is possible).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-13T16:47:35.990Z
Answers.Text: @nbro: Yes. I think the looser terminology is very common, because for many problems "basic" RNNs and LSTMs/GRUs are interchangeable - in the sense that they can have same inputs and outputs, so using LSTM or GRU is a hyperparameter choice. For some reason that is phrased as using one of (RNN, GRU, LSTM) to solve a problem as opposed to one of (Elman, GRU, LSTM). I am not sure why
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-13T15:57:44.440Z
Answers.LastActivityDate: 2021-09-13T16:06:27.260Z
Answers.LastEditDate: 2021-09-13T16:06:27.260Z
Answers.UserDisplayName: Neil Slater
Body: <p>Is the LSTM-Architecture a subcategory of RNNs? Or are they totally different?</p>
  <p>Literature doesn't seem to be unitary on this.
  <a href="http://mccormickml.com/assets/BERT/BERT_Mountain.png" rel="nofollow noreferrer">This figure</a> appears to explain the models to be alternatives, but I thought of them otherwise (LSTM to be a subcategory of RNN)</p>
  <p>LSTM as a subcategory of RNN is mentioned <a href="https://en.wikipedia.org/wiki/Long_short-term_memory" rel="nofollow noreferrer">in the Wikipedia article on LSTMs</a>:</p>
  <blockquote>
  <p>Long short-term memory (LSTM) is an artificial recurrent neural
  network (RNN) architecture...</p>
  </blockquote>
  
Tags: <terminology><recurrent-neural-networks><long-short-term-memory>
Title: Is LSTM a subcategory of RNN?
UserDisplayName: MrPlanck
ThreadText: # Is LSTM a subcategory of RNN?
  
  (Asked by: username_0 on 2021-09-13)
  
  Is the LSTM-Architecture a subcategory of RNNs? Or are they totally different?
  
  Literature doesn't seem to be unitary on this.
  [This figure](http://mccormickml.com/assets/BERT/BERT_Mountain.png) appears to explain the models to be alternatives, but I thought of them otherwise (LSTM to be a subcategory of RNN)
  
  LSTM as a subcategory of RNN is mentioned [in the Wikipedia article on LSTMs](https://en.wikipedia.org/wiki/Long_short-term_memory):
  
  > Long short-term memory (LSTM) is an artificial recurrent neural
  > network (RNN) architecture...
  
  **Tags:** terminology, recurrent-neural-networks, long-short-term-memory
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-09-13. Score: 10 | Accepted answer
  
  The Wikipedia article is more technically correct, in that the term RNN is formally taken to mean "a neural network with recurrent connections", and that includes many architectures that match this description, including LSTMs.
  
  However, it is also common to see "RNN" used as a short-hand for a kind of "Vanilla RNN" or "basic RNN", where one or more layers have weights connecting the layer to itself (its own activations from $t-1$ are concatenated to the external inputs at $t$), and there are no other gates or special combinations, just those recurrent connections.
  
  Oddly, this basic layer-based RNN archtecture is not listed in all the options on the [Wikipedia page on RNNs](https://en.wikipedia.org/wiki/Recurrent_neural_network) - probably the closest are Elman networks and Jordan networks which are ways to implement the recurrent connection. It is a valid architecture choice, and can be effective. The LSTM and GRU architectures improve on it in terms of handling longer sequences and preserving important signals over them when training (e.g. matching a starting and ending quote in text processing).
  
  > **Comments:**
  > **username_2**: So, I would clarify that this is a matter of terminology. If you define RNN (a short for recurrent neural networks) as the set of all neural networks with recurrent connections, then LSTMs could be considered a subset of RNN. However, if you define RNN as the vanilla RNN, then LSTMs are not a subset of RNNs (because vanilla RNNs do not have gates): it probably would be more the other way around (it may be possible to represent vanilla RNNs as LSTMs, though I don't remember the details now, so I am not really sure if this is possible).  
  > **username_1**: @username_2: Yes. I think the looser terminology is very common, because for many problems "basic" RNNs and LSTMs/GRUs are interchangeable - in the sense that they can have same inputs and outputs, so using LSTM or GRU is a hyperparameter choice. For some reason that is phrased as using one of (RNN, GRU, LSTM) to solve a problem as opposed to one of (Elman, GRU, LSTM). I am not sure why  
  
  ---
  
  