Answers.Body: <p>Just to be 100% sure - the diagram you refer to is a diagram of an LSTM CELL, not NETWORK. The operands you see on the diagram are operations within a cell, not separate "neurons". I think it is quite obvious, however reading your questions I just wanted to be 100% sure we are on the same page.</p>
  
  <p>Now, about layers. RNN networks (LSTM in particular) are just like any other ANN structure. Theoretically, a 1-hidden layer network can do any computation of a "deeper" network. ANN is a universal approximate of math functions. Still, multi-layer ANNs typically work better on more complex problems. Multi-layer typically needs less total connections, learns better and is less resource-demanding.</p>
  
  <p>In particular, multi-layer LSTMs are believed to be better at determining complex in-time patterns. I think there is no rigorous proof for this, however.
  Also, in practical applications - I did not see much improvement in network capabilities by adding additional LSTM layers. Adding more dense layers before/ after LSTM seemed to have a much better effect.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-02T09:02:51.330Z
Answers.LastActivityDate: 2018-09-02T09:02:51.330Z
Answers.UserDisplayName: Broono
Answers.Body: <p>(1) Yes this is the diagram for a classical LSTM unit. Of cause there are some variants and those diagrams would look slightly different.</p>
  
  <p>(2) It is very common for researchers to use more than one layers of LSTM and achieves better performance than a single layer one. A common way to "stack" LSTMs is to use the previous layer's output ($h_t$ in your diagram) as the input to the next layer ($x_t$). However, I have seldom seen any successful application of 5+ layers of LSTMs, while for CNNs it is common to use tens or even hundreds of layers.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-01T21:05:32.390Z
Answers.LastActivityDate: 2018-09-01T21:05:32.390Z
Answers.UserDisplayName: user12075
Body: <p>I have been researching LSTM neural networks.  I have seen <a href="https://cdn-images-1.medium.com/max/1600/1*Niu_c_FhGtLuHjrStkB_4Q.png" rel="nofollow noreferrer">this</a> diagram a lot and I have few questions about it. Firstly, is this diagram used for most LSTM neural networks?  </p>
  
  <p>Secondly, if it is, wouldn't only having single layers reduce it's usefulness?  </p>
  
Tags: <neural-networks><machine-learning><long-short-term-memory>
Title: Need Help With LSTM Neural Networks
UserDisplayName: Ryan B
ThreadText: # Need Help With LSTM Neural Networks
  
  (Asked by: username_0 on 2018-09-01)
  
  I have been researching LSTM neural networks. I have seen [this](https://cdn-images-1.medium.com/max/1600/1*Niu_c_FhGtLuHjrStkB_4Q.png) diagram a lot and I have few questions about it. Firstly, is this diagram used for most LSTM neural networks?
  
  Secondly, if it is, wouldn't only having single layers reduce it's usefulness?
  
  **Tags:** neural-networks, machine-learning, long-short-term-memory
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-09-02. Score: 0 | Accepted answer
  
  Just to be 100% sure - the diagram you refer to is a diagram of an LSTM CELL, not NETWORK. The operands you see on the diagram are operations within a cell, not separate "neurons". I think it is quite obvious, however reading your questions I just wanted to be 100% sure we are on the same page.
  
  Now, about layers. RNN networks (LSTM in particular) are just like any other ANN structure. Theoretically, a 1-hidden layer network can do any computation of a "deeper" network. ANN is a universal approximate of math functions. Still, multi-layer ANNs typically work better on more complex problems. Multi-layer typically needs less total connections, learns better and is less resource-demanding.
  
  In particular, multi-layer LSTMs are believed to be better at determining complex in-time patterns. I think there is no rigorous proof for this, however.
  Also, in practical applications - I did not see much improvement in network capabilities by adding additional LSTM layers. Adding more dense layers before/ after LSTM seemed to have a much better effect.
  
  
  ---
  
  ### Answer by username_2 on 2018-09-01. Score: 1
  
  (1) Yes this is the diagram for a classical LSTM unit. Of cause there are some variants and those diagrams would look slightly different.
  
  (2) It is very common for researchers to use more than one layers of LSTM and achieves better performance than a single layer one. A common way to "stack" LSTMs is to use the previous layer's output ($h\_t$ in your diagram) as the input to the next layer ($x\_t$). However, I have seldom seen any successful application of 5+ layers of LSTMs, while for CNNs it is common to use tens or even hundreds of layers.
  
  
  ---
  
  