Answers.Body: <p>In Keras, what you specify is the hidden layer size. So :</p>
  
  <pre><code>LSTM(128)
  </code></pre>
  
  <p>gives you a Keras layer representing a LSTM with a hidden layer size of 128.</p>
  
  <p>As you said :</p>
  
  <blockquote>
    <p>From the LSTM/RNN unfolding image or description, I found that each RNN cell take one time step at a time</p>
  </blockquote>
  
  <p>So if you picture your RNN for one time step, it will look like this : </p>
  
  <p><a href="https://i.sstatic.net/zbbpu.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zbbpu.png" alt="enter image description here"></a></p>
  
  <p>And if you unfold it <strong>in time</strong>, it look like this : </p>
  
  <p><a href="https://i.sstatic.net/0yUiz.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0yUiz.png" alt="enter image description here"></a></p>
  
  <p>You are not limited in your sequence size, this is one of the feature of RNN : since you input your sequence element by element, the size of the sequence can be variable.  </p>
  
  <p>That number, 128, represent just the size of the hidden layer of your LSTM. You can see the hidden layer of the LSTM as the memory of the RNN. </p>
  
  <p>Of course the goal is not for the LSTM to remember everything of the sequence, just link between elements. That's why the size of the hidden layer can be smaller than the size of your sequence.</p>
  
  <p>Sources :</p>
  
  <ul>
  <li><a href="https://keras.io/layers/recurrent/#lstm" rel="nofollow noreferrer">Keras documentation</a></li>
  <li><a href="http://adventuresinmachinelearning.com/keras-lstm-tutorial/" rel="nofollow noreferrer">This blog</a></li>
  </ul>
  
  <h2>Edit</h2>
  
  <p>From <a href="https://skymind.ai/wiki/lstm#long" rel="nofollow noreferrer">this blog</a> :</p>
  
  <blockquote>
    <p>The larger the network, the more powerful, but it’s also easier to overfit. Don’t want to try to learn a million parameters from 10,000 examples – parameters > examples = trouble.</p>
  </blockquote>
  
  <p>So the consequence of reducing the size of hidden state of LSTM is that it will be simpler. Might not be able to get the links between the element of the sequence. But if you put a too big size, your network will overfit ! And you absolutely don't want that.</p>
  
  <p>Another really good blog on LSTM : <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/" rel="nofollow noreferrer">this link</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-15T13:47:29.483Z
Answers.Text: Thank you for your answer. What if I lower the size of the hidden layer? say LSTM(64). What will be the consequences? Does it means lstm model has lower memory to capture long-term dependencies?
Answers.UserDisplayName: Bloodstone Programmer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-15T23:32:58.550Z
Answers.Text: I edited the answer :)
Answers.UserDisplayName: Astariul
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-15T08:31:44.950Z
Answers.LastActivityDate: 2018-10-15T23:32:40.617Z
Answers.LastEditDate: 2018-10-15T23:32:40.617Z
Answers.UserDisplayName: Astariul
Answers.Body: <p>Although this question has been answered I'd add a couple of remarks towards general neural networks design</p>
  
  <p>As you know very NN has three types of layers: input, hidden, and output. Once this network is initialized, you can iteratively tune the configuration during training.</p>
  
  <p>To optimize the network configuration we can use pruning.</p>
  
  <p>Pruning describes a set of techniques to trim network size (by nodes not layers) to improve computational performance and sometimes resolution performance. The gist of these techniques is removing nodes from the network during training by identifying those nodes which, if removed from the network, would not noticeably affect network performance (i.e., resolution of the data). (Even without using a formal pruning technique, you can get a rough idea of which nodes are not important by looking at your weight matrix after training; look weights very close to zero--it's the nodes on either end of those weights that are often removed during pruning.)</p>
  
  <p>You can find more here: <a href="https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw">https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-16T07:22:32.797Z
Answers.LastActivityDate: 2018-10-16T07:22:32.797Z
Answers.UserDisplayName: Daniel Chepenko
Body: <p>In keras, when we use an LSTM/RNN model, we need to specify the node [i.e., LSTM(128)]. I have a doubt regarding how it actually works. From the LSTM/RNN unfolding image or description, I found that each RNN cell take one time step at a time. What if my sequence is larger than 128? How to interpret this? Can anyone please explain me? Thank in advance.</p>
  
Tags: <machine-learning><recurrent-neural-networks><long-short-term-memory>
Title: What should I do when I have a variable-length sequence when instantiating an LSTM in Keras?
UserDisplayName: Bloodstone Programmer
ThreadText: # What should I do when I have a variable-length sequence when instantiating an LSTM in Keras?
  
  (Asked by: username_0 on 2018-10-15)
  
  In keras, when we use an LSTM/RNN model, we need to specify the node [i.e., LSTM(128)]. I have a doubt regarding how it actually works. From the LSTM/RNN unfolding image or description, I found that each RNN cell take one time step at a time. What if my sequence is larger than 128? How to interpret this? Can anyone please explain me? Thank in advance.
  
  **Tags:** machine-learning, recurrent-neural-networks, long-short-term-memory
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-10-15. Score: 1 | Accepted answer
  
  In Keras, what you specify is the hidden layer size. So :
  
  ```
  LSTM(128)
  
  ```
  
  gives you a Keras layer representing a LSTM with a hidden layer size of 128.
  
  As you said :
  
  > From the LSTM/RNN unfolding image or description, I found that each RNN cell take one time step at a time
  
  So if you picture your RNN for one time step, it will look like this :
  
  [![enter image description here](https://i.sstatic.net/zbbpu.png)](https://i.sstatic.net/zbbpu.png)
  
  And if you unfold it **in time**, it look like this :
  
  [![enter image description here](https://i.sstatic.net/0yUiz.png)](https://i.sstatic.net/0yUiz.png)
  
  You are not limited in your sequence size, this is one of the feature of RNN : since you input your sequence element by element, the size of the sequence can be variable.
  
  That number, 128, represent just the size of the hidden layer of your LSTM. You can see the hidden layer of the LSTM as the memory of the RNN.
  
  Of course the goal is not for the LSTM to remember everything of the sequence, just link between elements. That's why the size of the hidden layer can be smaller than the size of your sequence.
  
  Sources :
  
  * [Keras documentation](https://keras.io/layers/recurrent/#lstm)
  * [This blog](http://adventuresinmachinelearning.com/keras-lstm-tutorial/)
  
  ## Edit
  
  From [this blog](https://skymind.ai/wiki/lstm#long) :
  
  > The larger the network, the more powerful, but it’s also easier to overfit. Don’t want to try to learn a million parameters from 10,000 examples – parameters > examples = trouble.
  
  So the consequence of reducing the size of hidden state of LSTM is that it will be simpler. Might not be able to get the links between the element of the sequence. But if you put a too big size, your network will overfit ! And you absolutely don't want that.
  
  Another really good blog on LSTM : [this link](http://colah.github.io/posts/2015-08-Understanding-LSTMs/)
  
  > **Comments:**
  > **username_0**: Thank you for your answer. What if I lower the size of the hidden layer? say LSTM(64). What will be the consequences? Does it means lstm model has lower memory to capture long-term dependencies?  
  > **username_1**: I edited the answer :)  
  
  ---
  
  ### Answer by username_2 on 2018-10-16. Score: 0
  
  Although this question has been answered I'd add a couple of remarks towards general neural networks design
  
  As you know very NN has three types of layers: input, hidden, and output. Once this network is initialized, you can iteratively tune the configuration during training.
  
  To optimize the network configuration we can use pruning.
  
  Pruning describes a set of techniques to trim network size (by nodes not layers) to improve computational performance and sometimes resolution performance. The gist of these techniques is removing nodes from the network during training by identifying those nodes which, if removed from the network, would not noticeably affect network performance (i.e., resolution of the data). (Even without using a formal pruning technique, you can get a rough idea of which nodes are not important by looking at your weight matrix after training; look weights very close to zero--it's the nodes on either end of those weights that are often removed during pruning.)
  
  You can find more here: <https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw>
  
  
  ---
  
  