Answers.Body: <p>In the paper, <a href="https://arxiv.org/pdf/1503.04069.pdf" rel="nofollow noreferrer">LSTM: A Search Space Odyssey</a> (2017), by Klaus Greff et al., eight LSTM variants on three representative tasks (<a href="https://en.wikipedia.org/wiki/Speech_recognition" rel="nofollow noreferrer">speech recognition</a>, <a href="https://en.wikipedia.org/wiki/Handwriting_recognition" rel="nofollow noreferrer">handwriting recognition</a>, and <a href="https://arxiv.org/pdf/1206.6392.pdf" rel="nofollow noreferrer">polyphonic music modeling</a>) are compared. </p>
  
  <p>The compared variants are </p>
  
  <ol>
  <li><p><a href="ftp://ftp.idsia.ch/pub/juergen/nn_2005.pdf" rel="nofollow noreferrer">Vanilla LSTM</a> features three gates (input, forget, output), block input, a single cell, an output activation function, and <a href="https://pdfs.semanticscholar.org/e10f/98b86797ebf6c8caea6f54cacbc5a50e8b34.pdf" rel="nofollow noreferrer">peephole connections</a> (connections from the cell to the gates). The output of the block is recurrently connected back to the block input and all of the gates. The <a href="ftp://ftp.idsia.ch/pub/juergen/nn_2005.pdf" rel="nofollow noreferrer">vanilla LSTM</a> is trained using gradient descent and <a href="http://axon.cs.byu.edu/~martinez/classes/678/Papers/Werbos_BPTT.pdf" rel="nofollow noreferrer">back-propagation through time</a> (BPTT). The <a href="http://www.bioinf.jku.at/publications/older/2604.pdf" rel="nofollow noreferrer">original LSTM</a> (which is not the vanilla LSTM) does not contain, for example, the forget gate or the peephole connections (but the cell possesses a <em>constant error carousel</em>, a constant weight of <span class="math-container">$1$</span>). </p></li>
  <li><p>LSTM trained based on the <a href="https://en.wikipedia.org/wiki/Extended_Kalman_filter" rel="nofollow noreferrer"><em>decoupled extended Kalman filtering</em></a> <a href="https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2002-211.pdf" rel="nofollow noreferrer">(DEKF-LSTM)</a>, which enables the LSTM to be trained on some pathological cases at the cost of high computational complexity.</p></li>
  <li><p><a href="http://people.idsia.ch/~tino/papers/schmidhuber.neco07.pdf" rel="nofollow noreferrer">Vanilla LSTM trained with an evolution-based method (called <em>evolino</em>)</a>, instead of BPTT.</p></li>
  <li><p><a href="http://people.idsia.ch/~juergen/icann2009bayer.pdf" rel="nofollow noreferrer">LSTM block architectures evolved with a multi-objective evolutionary algorithm</a>, so that to maximize fitness on <a href="https://en.wikipedia.org/wiki/Context-sensitive_grammar" rel="nofollow noreferrer">context-sensitive grammar</a>.</p></li>
  <li><p><a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43905.pdf" rel="nofollow noreferrer">LSTM architectures for large scale acoustic modeling</a>, which introduces a <em>linear projection layer</em> that projects the output of the LSTM layer down before recurrent and forward connections in order to reduce the number of parameters for LSTM networks with many blocks.</p></li>
  <li><p>An <a href="https://ieeexplore.ieee.org/abstract/document/6981033" rel="nofollow noreferrer">LSTM architecture with a trainable scaling parameter for the slope of the gate activation functions</a>, which improves the performance of LSTM on an offline handwriting recognition dataset.</p></li>
  <li><p><a href="https://link.springer.com/chapter/10.1007%2F978-3-319-11179-7_1" rel="nofollow noreferrer">Dynamic Cortex Memory</a>, an LSTM composed of recurrent connections between the gates of a single block, but not between different blocks, which improves the convergence speed of LSTM.</p></li>
  <li><p><a href="https://arxiv.org/abs/1406.1078" rel="nofollow noreferrer">Gated Recurrent Unit (GRU)</a>, which simplifies the architecture of the LSTM by combining the input and forget gate into an <em>update gate</em>.</p></li>
  </ol>
  
  <p>There are other related neural network architectures, such as the <a href="https://arxiv.org/pdf/1410.5401.pdf" rel="nofollow noreferrer">neural Turing machine (NTM)</a> or <a href="https://www.nature.com/articles/nature20101" rel="nofollow noreferrer">differentiable neural computer (DNC)</a>. In general, there are several architectures that use LSTM blocks, even though they are not just recurrent neural networks. Other examples are the <a href="https://arxiv.org/abs/1511.06279" rel="nofollow noreferrer">neural programmer-interpreter (NPI)</a> or the <a href="https://arxiv.org/pdf/1705.02670.pdf" rel="nofollow noreferrer">meta-controller</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-04T11:04:53.203Z
Answers.LastActivityDate: 2019-09-05T12:07:45.953Z
Answers.LastEditDate: 2019-09-05T12:07:45.953Z
Answers.UserDisplayName: nbro
Body: <p>I've been doing some class assignments recently on building various neural networks. For convolutional networks, there are several well-known architectures such as LeNet, VGG etc. Such "classic" models are frequently referenced as starting points when building new CNNs. </p>
  
  <p>Are there similar examples for RNN/LSTM networks? All I've found so far are articles and slides explaining recurrent neurons, LSTM layers, and the math behind them, but no well-known examples of entire multi-layered network architectures, unlike CNNs which seem to have in abundance.</p>
  
Tags: <neural-networks><recurrent-neural-networks><long-short-term-memory>
Title: What are some examples of LSTM architectures?
UserDisplayName: thegreatjedi
ThreadText: # What are some examples of LSTM architectures?
  
  (Asked by: username_0 on 2019-09-04)
  
  I've been doing some class assignments recently on building various neural networks. For convolutional networks, there are several well-known architectures such as LeNet, VGG etc. Such "classic" models are frequently referenced as starting points when building new CNNs.
  
  Are there similar examples for RNN/LSTM networks? All I've found so far are articles and slides explaining recurrent neurons, LSTM layers, and the math behind them, but no well-known examples of entire multi-layered network architectures, unlike CNNs which seem to have in abundance.
  
  **Tags:** neural-networks, recurrent-neural-networks, long-short-term-memory
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-09-04. Score: 2
  
  In the paper, [LSTM: A Search Space Odyssey](https://arxiv.org/pdf/1503.04069.pdf) (2017), by Klaus Greff et al., eight LSTM variants on three representative tasks ([speech recognition](https://en.wikipedia.org/wiki/Speech_recognition), [handwriting recognition](https://en.wikipedia.org/wiki/Handwriting_recognition), and [polyphonic music modeling](https://arxiv.org/pdf/1206.6392.pdf)) are compared.
  
  The compared variants are
  
  1. [Vanilla LSTM](ftp://ftp.idsia.ch/pub/juergen/nn_2005.pdf) features three gates (input, forget, output), block input, a single cell, an output activation function, and [peephole connections](https://pdfs.semanticscholar.org/e10f/98b86797ebf6c8caea6f54cacbc5a50e8b34.pdf) (connections from the cell to the gates). The output of the block is recurrently connected back to the block input and all of the gates. The [vanilla LSTM](ftp://ftp.idsia.ch/pub/juergen/nn_2005.pdf) is trained using gradient descent and [back-propagation through time](http://axon.cs.byu.edu/~martinez/classes/678/Papers/Werbos_BPTT.pdf) (BPTT). The [original LSTM](http://www.bioinf.jku.at/publications/older/2604.pdf) (which is not the vanilla LSTM) does not contain, for example, the forget gate or the peephole connections (but the cell possesses a *constant error carousel*, a constant weight of $1$).
  2. LSTM trained based on the [*decoupled extended Kalman filtering*](https://en.wikipedia.org/wiki/Extended_Kalman_filter) [(DEKF-LSTM)](https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2002-211.pdf), which enables the LSTM to be trained on some pathological cases at the cost of high computational complexity.
  3. [Vanilla LSTM trained with an evolution-based method (called *evolino*)](http://people.idsia.ch/~tino/papers/schmidhuber.neco07.pdf), instead of BPTT.
  4. [LSTM block architectures evolved with a multi-objective evolutionary algorithm](http://people.idsia.ch/~juergen/icann2009bayer.pdf), so that to maximize fitness on [context-sensitive grammar](https://en.wikipedia.org/wiki/Context-sensitive_grammar).
  5. [LSTM architectures for large scale acoustic modeling](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43905.pdf), which introduces a *linear projection layer* that projects the output of the LSTM layer down before recurrent and forward connections in order to reduce the number of parameters for LSTM networks with many blocks.
  6. An [LSTM architecture with a trainable scaling parameter for the slope of the gate activation functions](https://ieeexplore.ieee.org/abstract/document/6981033), which improves the performance of LSTM on an offline handwriting recognition dataset.
  7. [Dynamic Cortex Memory](https://link.springer.com/chapter/10.1007%2F978-3-319-11179-7_1), an LSTM composed of recurrent connections between the gates of a single block, but not between different blocks, which improves the convergence speed of LSTM.
  8. [Gated Recurrent Unit (GRU)](https://arxiv.org/abs/1406.1078), which simplifies the architecture of the LSTM by combining the input and forget gate into an *update gate*.
  
  There are other related neural network architectures, such as the [neural Turing machine (NTM)](https://arxiv.org/pdf/1410.5401.pdf) or [differentiable neural computer (DNC)](https://www.nature.com/articles/nature20101). In general, there are several architectures that use LSTM blocks, even though they are not just recurrent neural networks. Other examples are the [neural programmer-interpreter (NPI)](https://arxiv.org/abs/1511.06279) or the [meta-controller](https://arxiv.org/pdf/1705.02670.pdf).
  
  
  ---
  
  