Answers.Body: <p>Your question is quite broad, but here are some tips.</p>
  
  <p>Specifically for LSTMs, see this Reddit discussion <a href="https://www.reddit.com/r/MachineLearning/comments/4behuh/does_the_number_of_layers_in_an_lstm_network/" rel="noreferrer">Does the number of layers in an LSTM network affect its ability to remember long patterns?</a></p>
  
  <p>The main point is that there is usually no rule for the number of hidden nodes you should use, it is something you have to figure out for each case by trial and error.</p>
  
  <p>If you are also interested in feedforward networks, see the question <a href="https://stats.stackexchange.com/q/181/82135">How to choose the number of hidden layers and nodes in a feedforward neural network?</a> at Stats SE. Specifically, <a href="https://stats.stackexchange.com/a/136542/82135">this answer</a> was helpful.</p>
  
  <blockquote>
    <p>There's one additional rule of thumb that helps for supervised learning problems. You can usually prevent over-fitting if you keep your number of neurons below:</p>
    
    <p><span class="math-container">$$N_h = \frac{N_s} {(\alpha * (N_i + N_o))}$$</span></p>
    
    <ul>
    <li><span class="math-container">$N_i$</span> = number of input neurons.   </li>
    <li><span class="math-container">$N_o$</span> = number of output neurons.  </li>
    <li><span class="math-container">$N_s$</span> = number of samples in training data set.  </li>
    <li><span class="math-container">$\alpha$</span> = an arbitrary scaling factor usually 2-10.  </li>
    </ul>
    
    <p><a href="http://www.solver.com/training-artificial-neural-network-intro" rel="noreferrer">Others recommend</a> setting <span class="math-container">$alpha$</span> to a value between 5 and 10, but I find a value of 2 will often work without overfitting. You can think of alpha as the effective branching factor or number of nonzero weights for each neuron. Dropout layers will bring the "effective" branching factor way down from the actual mean branching factor for your network.</p>
    
    <p>As explained by this <a href="http://hagan.okstate.edu/NNDesign.pdf#page=469" rel="noreferrer">excellent NN Design text</a>, you want to limit the number of free parameters in your model (i.e. its <a href="https://stats.stackexchange.com/q/57027/15974">degree</a> or the number of nonzero weights) to a small portion of the degrees of freedom in your data. The degrees of freedom in your data is the number samples * degrees of freedom (dimensions) in each sample or <span class="math-container">$N_s * (N_i + N_o)$</span> (assuming they're all independent). So <span class="math-container">$\alpha$</span> is a way to indicate how general you want your model to be, or how much you want to prevent overfitting.</p>
    
    <p>For an automated procedure you'd start with an alpha of 2 (twice as many degrees of freedom in your training data as your model) and work your way up to 10 if the error (loss) for your training dataset is significantly smaller than for your test dataset.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-06T14:52:36.220Z
Answers.Text: and what is 'sample' here, number of minibatches or the number of individual time sequence entry?
Answers.UserDisplayName: Ilya Yevlampiev
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-04-14T15:55:12.433Z
Answers.LastActivityDate: 2020-03-07T16:16:51.233Z
Answers.LastEditDate: 2020-03-07T16:16:51.233Z
Answers.UserDisplayName: Thomas Wagenaar
Answers.Body: <p>The selection of the number of hidden layers and the number of memory cells in LSTM probably depends on the application domain and context where you want to apply this LSTM. </p>
  
  <p>The optimal number of hidden units could be smaller than the number of inputs. AFAIK, there is no rule like multiply the number of inputs with <span class="math-container">$N$</span>. If you have a lot of training examples, you can use multiple hidden units, but sometimes just 2 hidden units work best with little data. </p>
  
  <p>Usually, people use one hidden layer for simple tasks, but nowadays research in deep neural network architectures show that many hidden layers can be fruitful for a difficult object, handwritten character, and face recognition problems.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-06-26T07:43:22.363Z
Answers.LastActivityDate: 2020-03-07T15:42:43.753Z
Answers.LastEditDate: 2020-03-07T15:42:43.753Z
Answers.UserDisplayName: Maheshwar Ligade
Answers.Body: <p>Have a look at the paper <a href="https://wiki.inf.ed.ac.uk/twiki/pub/CSTR/ListenTerm1201415/sak2.pdf" rel="nofollow noreferrer">Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling</a> (2014), where different LSTM architectures are compared. In the abstract, the authors write the following.</p>
  
  <blockquote>
    <p>We show that a two-layer deep LSTM RNN where each LSTM layer has a linear recurrent projection layer can exceed state-of-the-art speech recognition performance</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-04T19:58:50.787Z
Answers.LastActivityDate: 2020-03-07T15:50:04.103Z
Answers.LastEditDate: 2020-03-07T15:50:04.103Z
Answers.UserDisplayName: Dieshe
Answers.Body: <p>In general, there are no guidelines on how to determine the number of layers or the number of memory cells in an LSTM.</p>
  
  <p>The number of layers and cells required in an LSTM might depend on several aspects of the problem:</p>
  
  <ol>
  <li><p>The complexity of the dataset, such as the number of features, the number of data points, etc. </p></li>
  <li><p>The data-generating process. For example, the prediction of oil prices compared to the prediction of GDP is a well-understood economy. The latter is much easier than the former. Thus, predicting oil prices might require more LSTM memory cells to predict, with the same accuracy, as compared to the GDP. </p></li>
  <li><p>The accuracy required for the use case. The number of memory cells will <strong>heavily</strong> depend on this. If the goal is to beat the state-of-the-art model, in general, one needs more LSTM cells. Compare that to the goal of coming up with a reasonable prediction, which would need fewer LSTM cells.</p></li>
  </ol>
  
  <p>I follow these steps when modeling using LSTM.</p>
  
  <ol>
  <li><p>Try a single hidden layer with 2 or 3 memory cells. See how it performs against a benchmark. If it is a time series problem, then I generally make a forecast from classical time series techniques as benchmark. </p></li>
  <li><p>Try and increase the number of memory cells. If the performance is not increasing much then move on to the next step.</p></li>
  <li><p>Start making the network deeper, i.e. add another layer with a small number of memory cells. </p></li>
  </ol>
  
  <p>As a side note, there is no limit to the amount of labor that can be devoted to reach that global minimum of the loss function and tune the best hyper-parameters. So, having the focus on the end goal for modeling should be the strategy rather than trying to increase the accuracy as much as possible.</p>
  
  <p>Most of the problems can be handled using 2-3 layers of the network.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-05T14:59:31.667Z
Answers.LastActivityDate: 2020-03-07T16:08:05.407Z
Answers.LastEditDate: 2020-03-07T16:08:05.407Z
Answers.UserDisplayName: naive
Body: <p>I am trying to find some existing research on how to select the number of hidden layers and the size of these of an LSTM-based RNN.</p>
  <p>Is there an article where this problem is being investigated, i.e., how many memory cells should one use? I assume it totaly depends on the application and in which context the model is being used, but what does the research say?</p>
  
Tags: <reference-request><recurrent-neural-networks><long-short-term-memory><hyperparameter-optimization><hyper-parameters>
Title: How to select number of hidden layers and number of memory cells in an LSTM?
UserDisplayName: Stephen Johnson
ThreadText: # How to select number of hidden layers and number of memory cells in an LSTM?
  
  (Asked by: username_0 on 2017-04-14)
  
  I am trying to find some existing research on how to select the number of hidden layers and the size of these of an LSTM-based RNN.
  
  Is there an article where this problem is being investigated, i.e., how many memory cells should one use? I assume it totaly depends on the application and in which context the model is being used, but what does the research say?
  
  **Tags:** reference-request, recurrent-neural-networks, long-short-term-memory, hyperparameter-optimization, hyper-parameters
  
  ---
  
  ## 4 answers
  
  ### Answer by username_1 on 2017-04-14. Score: 37
  
  Your question is quite broad, but here are some tips.
  
  Specifically for LSTMs, see this Reddit discussion [Does the number of layers in an LSTM network affect its ability to remember long patterns?](https://www.reddit.com/r/MachineLearning/comments/4behuh/does_the_number_of_layers_in_an_lstm_network/)
  
  The main point is that there is usually no rule for the number of hidden nodes you should use, it is something you have to figure out for each case by trial and error.
  
  If you are also interested in feedforward networks, see the question [How to choose the number of hidden layers and nodes in a feedforward neural network?](https://stats.stackexchange.com/q/181/82135) at Stats SE. Specifically, [this answer](https://stats.stackexchange.com/a/136542/82135) was helpful.
  
  > There's one additional rule of thumb that helps for supervised learning problems. You can usually prevent over-fitting if you keep your number of neurons below:
  >
  > $$N\_h = \frac{N\_s} {(\alpha \* (N\_i + N\_o))}$$
  >
  > * $N\_i$ = number of input neurons.
  > * $N\_o$ = number of output neurons.
  > * $N\_s$ = number of samples in training data set.
  > * $\alpha$ = an arbitrary scaling factor usually 2-10.
  >
  > [Others recommend](http://www.solver.com/training-artificial-neural-network-intro) setting $alpha$ to a value between 5 and 10, but I find a value of 2 will often work without overfitting. You can think of alpha as the effective branching factor or number of nonzero weights for each neuron. Dropout layers will bring the "effective" branching factor way down from the actual mean branching factor for your network.
  >
  > As explained by this [excellent NN Design text](http://hagan.okstate.edu/NNDesign.pdf#page=469), you want to limit the number of free parameters in your model (i.e. its [degree](https://stats.stackexchange.com/q/57027/15974) or the number of nonzero weights) to a small portion of the degrees of freedom in your data. The degrees of freedom in your data is the number samples \* degrees of freedom (dimensions) in each sample or $N\_s \* (N\_i + N\_o)$ (assuming they're all independent). So $\alpha$ is a way to indicate how general you want your model to be, or how much you want to prevent overfitting.
  >
  > For an automated procedure you'd start with an alpha of 2 (twice as many degrees of freedom in your training data as your model) and work your way up to 10 if the error (loss) for your training dataset is significantly smaller than for your test dataset.
  
  > **Comments:**
  > **username_2**: and what is 'sample' here, number of minibatches or the number of individual time sequence entry?  
  
  ---
  
  ### Answer by username_3 on 2017-06-26. Score: 9
  
  The selection of the number of hidden layers and the number of memory cells in LSTM probably depends on the application domain and context where you want to apply this LSTM.
  
  The optimal number of hidden units could be smaller than the number of inputs. AFAIK, there is no rule like multiply the number of inputs with $N$. If you have a lot of training examples, you can use multiple hidden units, but sometimes just 2 hidden units work best with little data.
  
  Usually, people use one hidden layer for simple tasks, but nowadays research in deep neural network architectures show that many hidden layers can be fruitful for a difficult object, handwritten character, and face recognition problems.
  
  
  ---
  
  ### Answer by username_5 on 2018-07-05. Score: 7
  
  In general, there are no guidelines on how to determine the number of layers or the number of memory cells in an LSTM.
  
  The number of layers and cells required in an LSTM might depend on several aspects of the problem:
  
  1. The complexity of the dataset, such as the number of features, the number of data points, etc.
  2. The data-generating process. For example, the prediction of oil prices compared to the prediction of GDP is a well-understood economy. The latter is much easier than the former. Thus, predicting oil prices might require more LSTM memory cells to predict, with the same accuracy, as compared to the GDP.
  3. The accuracy required for the use case. The number of memory cells will **heavily** depend on this. If the goal is to beat the state-of-the-art model, in general, one needs more LSTM cells. Compare that to the goal of coming up with a reasonable prediction, which would need fewer LSTM cells.
  
  I follow these steps when modeling using LSTM.
  
  1. Try a single hidden layer with 2 or 3 memory cells. See how it performs against a benchmark. If it is a time series problem, then I generally make a forecast from classical time series techniques as benchmark.
  2. Try and increase the number of memory cells. If the performance is not increasing much then move on to the next step.
  3. Start making the network deeper, i.e. add another layer with a small number of memory cells.
  
  As a side note, there is no limit to the amount of labor that can be devoted to reach that global minimum of the loss function and tune the best hyper-parameters. So, having the focus on the end goal for modeling should be the strategy rather than trying to increase the accuracy as much as possible.
  
  Most of the problems can be handled using 2-3 layers of the network.
  
  
  ---
  
  ### Answer by username_4 on 2018-07-04. Score: 4
  
  Have a look at the paper [Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling](https://wiki.inf.ed.ac.uk/twiki/pub/CSTR/ListenTerm1201415/sak2.pdf) (2014), where different LSTM architectures are compared. In the abstract, the authors write the following.
  
  > We show that a two-layer deep LSTM RNN where each LSTM layer has a linear recurrent projection layer can exceed state-of-the-art speech recognition performance
  
  
  ---
  
  