Answers.Body: <p>From my personal experience, the <code>units</code> hyperparam in LSTM is not necessary to be the same as max sequence length. Add more units to have the loss curve dive faster.</p>
  <p>And about the number of LSTM layers, trying out a single LSTM layer is a good start point, the model trains better with more LSTM layers.</p>
  <p>For example, MAX_SEQ_LEN=10, in Keras:</p>
  <pre><code>Lstm1 = LSTM(units=30, return_sequences=True); #Time sequence, to feed to next layer
  Lstm2 = LSTM(units=20, return_sequences=True); #Time sequence, to feed to next layer
  Lstm3 = LSTM(units=10, return_sequences=False);
  
  Output = Lstm3(Lstm2(Lstm1));
  <span class="math-container">```</span>
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-18T01:19:14.920Z
Answers.LastActivityDate: 2020-07-18T01:19:14.920Z
Answers.UserDisplayName: Dan D.
Body: <p>Theoretically, number of units for a LSTM layer is the number of hidden states or the max length of sequences as per my practice.</p>
  <p>For example, in Keras:</p>
  <pre><code>Lstm1 = LSTM(units=MAX_SEQ_LEN, return_sequences=False);
  </code></pre>
  <p>However, with <strong>lots of sequences to train</strong>, should I add more LSTM layers? because increasing MAX_SEQ_LEN is not the way as it doesn't help make the network better since the extra number of hidden states isn't useful any more.</p>
  <p>I'm considering increasing number of LSTM layers, but <strong>how many are enough?</strong></p>
  <p>For example, 3 of them:</p>
  <pre><code>Lstm1 = LSTM(units=MAX_SEQ_LEN, return_sequences=True);
  Lstm2 = LSTM(units=MAX_SEQ_LEN, return_sequences=True);
  Lstm3 = LSTM(units=MAX_SEQ_LEN, return_sequences=False);
  </code></pre>
  
Tags: <keras><long-short-term-memory><sequence-modeling><hidden-layers><network-design>
Title: Number of LSTM layers needed to learn a certain number of sequences
UserDisplayName: Dan D.
ThreadText: # Number of LSTM layers needed to learn a certain number of sequences
  
  (Asked by: username_0 on 2020-07-06)
  
  Theoretically, number of units for a LSTM layer is the number of hidden states or the max length of sequences as per my practice.
  
  For example, in Keras:
  
  ```
  Lstm1 = LSTM(units=MAX_SEQ_LEN, return_sequences=False);
  
  ```
  
  However, with **lots of sequences to train**, should I add more LSTM layers? because increasing MAX\_SEQ\_LEN is not the way as it doesn't help make the network better since the extra number of hidden states isn't useful any more.
  
  I'm considering increasing number of LSTM layers, but **how many are enough?**
  
  For example, 3 of them:
  
  ```
  Lstm1 = LSTM(units=MAX_SEQ_LEN, return_sequences=True);
  Lstm2 = LSTM(units=MAX_SEQ_LEN, return_sequences=True);
  Lstm3 = LSTM(units=MAX_SEQ_LEN, return_sequences=False);
  
  ```
  
  **Tags:** keras, long-short-term-memory, sequence-modeling, hidden-layers, network-design
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-07-18. Score: 1 | Accepted answer
  
  From my personal experience, the `units` hyperparam in LSTM is not necessary to be the same as max sequence length. Add more units to have the loss curve dive faster.
  
  And about the number of LSTM layers, trying out a single LSTM layer is a good start point, the model trains better with more LSTM layers.
  
  For example, MAX\_SEQ\_LEN=10, in Keras:
  
  ```
  Lstm1 = LSTM(units=30, return_sequences=True); #Time sequence, to feed to next layer
  Lstm2 = LSTM(units=20, return_sequences=True); #Time sequence, to feed to next layer
  Lstm3 = LSTM(units=10, return_sequences=False);
  
  Output = Lstm3(Lstm2(Lstm1));
  ```
  
  ```
  
  
  ---
  
  