Body: <p>I watched a video explaining how LSTM cells have very rudimentary feed-forward neural networks, basically a 2 layer input-output with no hidden layers.</p>
  
  <p>Why don't LSTM cells have more complex neural networks before each gate, i.e. containing 1 hidden layer?</p>
  
  <p>I would think that if you want more advanced gating decisions, that you would use at least 1 hidden layer to have more complex processing.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-13T20:57:28.203Z
Comments.Text: What is the video? Maybe it was considering the nodes themselves, as you can have multiple layers of LSTM nodes without problem, try it :)
Comments.UserDisplayName: benbyford
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-14T02:46:59.160Z
Comments.Text: The reason is likely because the benefits of increasing the number of hidden units in an LSTM outweigh (or are essentially the same as) the benefits from increasing the depth before each node. Each of these increase computation time, so you should always chose the one that has the largest benefit. This conclusion was likely come to simply through experimentation. I can not find any papers on this topic however, but I would guess it's been done as a small segment of a paper and was probably quickly shown to have minimal or no benefit
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-15T15:14:49.063Z
Comments.Text: Mostly LSTM dealing with sequence of low-dimensional samples, like words (below 300 dims). For LSTM dealing with high-dimensional samples like images (much more then 10^3 dims) LSTM could be put on top of traditional feed-forward network like CNN
Comments.UserDisplayName: mirror2image
Tags: <deep-learning><long-short-term-memory>
Title: Why don't the neural networks inside LSTM cells contain hidden layers?
UserDisplayName: Snowybluesky
ThreadText: # Why don't the neural networks inside LSTM cells contain hidden layers?
  
  (Asked by: username_0 on 2020-01-13)
  
  I watched a video explaining how LSTM cells have very rudimentary feed-forward neural networks, basically a 2 layer input-output with no hidden layers.
  
  Why don't LSTM cells have more complex neural networks before each gate, i.e. containing 1 hidden layer?
  
  I would think that if you want more advanced gating decisions, that you would use at least 1 hidden layer to have more complex processing.
  
  **Tags:** deep-learning, long-short-term-memory
  > **Comments:**
  > **username_1**: What is the video? Maybe it was considering the nodes themselves, as you can have multiple layers of LSTM nodes without problem, try it :)  
  > **username_2**: The reason is likely because the benefits of increasing the number of hidden units in an LSTM outweigh (or are essentially the same as) the benefits from increasing the depth before each node. Each of these increase computation time, so you should always chose the one that has the largest benefit. This conclusion was likely come to simply through experimentation. I can not find any papers on this topic however, but I would guess it's been done as a small segment of a paper and was probably quickly shown to have minimal or no benefit  
  > **username_3**: Mostly LSTM dealing with sequence of low-dimensional samples, like words (below 300 dims). For LSTM dealing with high-dimensional samples like images (much more then 10^3 dims) LSTM could be put on top of traditional feed-forward network like CNN  
  
  ---
  
  ## 0 answers
  
  