Answers.Body: <h3>RNNs have recurrent connections and/or layers</h3>
  <p>You can describe a recurrent neural network (RNN) or a long short-term memory (LSTM), depending on the context, at different levels of abstraction. For example, you could say that an RNN is any neural network that contains one or more <strong>recurrent</strong> (or <strong>cyclic</strong>) <strong>connections</strong>. Or you could say that layer <span class="math-container">$l$</span> of neural network <span class="math-container">$N$</span> is a <strong>recurrent layer</strong>, given that it contains units (or neurons) with recurrent connections, but <span class="math-container">$N$</span> may not contain only recurrent layers (for example, it may also be composed of feedforward layers, i.e. layers with units that contain only feedforward connections).</p>
  <p>In any case, a recurrent neural network is almost always described as a neural network (NN) and not as a layer (this should also be obvious from the name).</p>
  <h3>LSTM can refer to a unit, layer or neural network</h3>
  <p>On the other hand, depending on the context, the term &quot;LSTM&quot; alone can refer to an</p>
  <ul>
  <li><strong>LSTM unit</strong> (or neuron),</li>
  <li>an <strong>LSTM layer</strong> (many LSTM units), or</li>
  <li>an <strong>LSTM neural network</strong> (a neural network with LSTM units or layers).</li>
  </ul>
  <p>People may also refer to neural networks with LSTM units as <strong>LSTMs</strong> (plural version of LSTM).</p>
  <h3>LSTMs are RNNs</h3>
  <p>An LSTM unit is a recurrent unit, that is, a unit (or neuron) that contains cyclic connections, so <strong>an LSTM neural network is a recurrent neural network (RNN)</strong>.</p>
  <h3>LSTM units/neurons</h3>
  <p>The main difference between an LSTM unit and a standard RNN unit is that the LSTM unit is <em>more sophisticated</em>. More precisely, it is composed of the so-called <strong>gates</strong> that supposedly regulate better the flow of information through the unit.</p>
  <p>Here's a typical representation (or diagram) of an LSTM (more precisely, an LSTM with a so-called <em>peephole connection</em>).</p>
  <p><a href="https://i.sstatic.net/ltQT0.png" rel="noreferrer"><img src="https://i.sstatic.net/ltQT0.png" alt="enter image description here" /></a></p>
  <p>This can actually represent both an LSTM unit (and, in that case, the variables are scalars) or an LSTM layer (and, in that case, the variables are vectors or matrices).</p>
  <p>You can see from this diagram that an LSTM unit (or layer) is composed of <strong>gates</strong>, denoted by</p>
  <ul>
  <li><span class="math-container">$i_t$</span> (the <strong>input gate</strong>: the gate that regulates the input into the unit/layer),</li>
  <li><span class="math-container">$o_t$</span> (the <strong>output gate</strong>: the gate that regulates the output from the unit)</li>
  <li><span class="math-container">$f_t$</span> (the <strong>forget gate</strong>: the gate that regulates what the <em>cell</em> should forget)</li>
  </ul>
  <p>and recurrent connections (e.g. the connection from the cell into the forget gate and vice-versa).</p>
  <p>It's also composed of a <strong>cell</strong>, which is the only thing that a neuron of a &quot;vanilla&quot; RNN contains.</p>
  <p>To understand the details (i.e. the purpose of all these components, such as the gates), you could read <a href="https://www.bioinf.jku.at/publications/older/2604.pdf" rel="noreferrer">the paper that originally proposed the LSTM</a> by S. Hochreiter and J. Schmidhuber. However, there may be other more accessible and understandable papers, articles or video lessons on the topic, which you can find on the web.</p>
  <h3>LSTMs also have recurrent connections!</h3>
  <p>Given the presence of cyclic connections, <strong>any</strong> recurrent neural network (either an LSTM or not) may be represented as a graph that contains one or more cyclic connections. For example, the following diagram may represent both a standard/vanilla RNN or an LSTM neural network (or maybe a variant of it, e.g. the <a href="https://en.wikipedia.org/wiki/Gated_recurrent_unit" rel="noreferrer">GRU</a>).</p>
  <p><a href="https://i.sstatic.net/XUmdH.png" rel="noreferrer"><img src="https://i.sstatic.net/XUmdH.png" alt="enter image description here" /></a></p>
  <h3>When should you use RNNs and LSTMs?</h3>
  <p>RNNs are particularly suited for tasks that involve sequences (thanks to the recurrent connections). For example, they are often used for machine translation, where the sequences are sentences or words. In practice, an LSTM is often used, as opposed to a vanilla (or standard) RNN, because it is more computationally effective. In fact, the LSTM was introduced to solve a problem that standard RNNs suffer from, i.e. the <a href="http://www.bioinf.jku.at/publications/older/2304.pdf" rel="noreferrer">vanishing gradient problem</a>. (Now, for these tasks, there are also the <a href="https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf" rel="noreferrer">transformers</a>, but the question was not about them).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-23T21:39:37.643Z
Answers.LastActivityDate: 2021-10-14T12:16:31.417Z
Answers.LastEditDate: 2021-10-14T12:16:31.417Z
Answers.UserDisplayName: nbro
Body: <p>What is the difference between LSTM and RNN? I know that RNN is a layer used in neural networks, but what exactly is an LSTM? Is it also a layer with the same characteristics?</p>
  
Tags: <neural-networks><comparison><recurrent-neural-networks><long-short-term-memory><recurrent-layers>
Title: What is the difference between LSTM and RNN?
UserDisplayName: Mao76
ThreadText: # What is the difference between LSTM and RNN?
  
  (Asked by: username_0 on 2020-02-23)
  
  What is the difference between LSTM and RNN? I know that RNN is a layer used in neural networks, but what exactly is an LSTM? Is it also a layer with the same characteristics?
  
  **Tags:** neural-networks, comparison, recurrent-neural-networks, long-short-term-memory, recurrent-layers
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-23. Score: 9 | Accepted answer
  
  ### RNNs have recurrent connections and/or layers
  
  You can describe a recurrent neural network (RNN) or a long short-term memory (LSTM), depending on the context, at different levels of abstraction. For example, you could say that an RNN is any neural network that contains one or more **recurrent** (or **cyclic**) **connections**. Or you could say that layer $l$ of neural network $N$ is a **recurrent layer**, given that it contains units (or neurons) with recurrent connections, but $N$ may not contain only recurrent layers (for example, it may also be composed of feedforward layers, i.e. layers with units that contain only feedforward connections).
  
  In any case, a recurrent neural network is almost always described as a neural network (NN) and not as a layer (this should also be obvious from the name).
  
  ### LSTM can refer to a unit, layer or neural network
  
  On the other hand, depending on the context, the term "LSTM" alone can refer to an
  
  * **LSTM unit** (or neuron),
  * an **LSTM layer** (many LSTM units), or
  * an **LSTM neural network** (a neural network with LSTM units or layers).
  
  People may also refer to neural networks with LSTM units as **LSTMs** (plural version of LSTM).
  
  ### LSTMs are RNNs
  
  An LSTM unit is a recurrent unit, that is, a unit (or neuron) that contains cyclic connections, so **an LSTM neural network is a recurrent neural network (RNN)**.
  
  ### LSTM units/neurons
  
  The main difference between an LSTM unit and a standard RNN unit is that the LSTM unit is *more sophisticated*. More precisely, it is composed of the so-called **gates** that supposedly regulate better the flow of information through the unit.
  
  Here's a typical representation (or diagram) of an LSTM (more precisely, an LSTM with a so-called *peephole connection*).
  
  [![enter image description here](https://i.sstatic.net/ltQT0.png)](https://i.sstatic.net/ltQT0.png)
  
  This can actually represent both an LSTM unit (and, in that case, the variables are scalars) or an LSTM layer (and, in that case, the variables are vectors or matrices).
  
  You can see from this diagram that an LSTM unit (or layer) is composed of **gates**, denoted by
  
  * $i\_t$ (the **input gate**: the gate that regulates the input into the unit/layer),
  * $o\_t$ (the **output gate**: the gate that regulates the output from the unit)
  * $f\_t$ (the **forget gate**: the gate that regulates what the *cell* should forget)
  
  and recurrent connections (e.g. the connection from the cell into the forget gate and vice-versa).
  
  It's also composed of a **cell**, which is the only thing that a neuron of a "vanilla" RNN contains.
  
  To understand the details (i.e. the purpose of all these components, such as the gates), you could read [the paper that originally proposed the LSTM](https://www.bioinf.jku.at/publications/older/2604.pdf) by S. Hochreiter and J. Schmidhuber. However, there may be other more accessible and understandable papers, articles or video lessons on the topic, which you can find on the web.
  
  ### LSTMs also have recurrent connections!
  
  Given the presence of cyclic connections, **any** recurrent neural network (either an LSTM or not) may be represented as a graph that contains one or more cyclic connections. For example, the following diagram may represent both a standard/vanilla RNN or an LSTM neural network (or maybe a variant of it, e.g. the [GRU](https://en.wikipedia.org/wiki/Gated_recurrent_unit)).
  
  [![enter image description here](https://i.sstatic.net/XUmdH.png)](https://i.sstatic.net/XUmdH.png)
  
  ### When should you use RNNs and LSTMs?
  
  RNNs are particularly suited for tasks that involve sequences (thanks to the recurrent connections). For example, they are often used for machine translation, where the sequences are sentences or words. In practice, an LSTM is often used, as opposed to a vanilla (or standard) RNN, because it is more computationally effective. In fact, the LSTM was introduced to solve a problem that standard RNNs suffer from, i.e. the [vanishing gradient problem](http://www.bioinf.jku.at/publications/older/2304.pdf). (Now, for these tasks, there are also the [transformers](https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf), but the question was not about them).
  
  
  ---
  
  