Answers.Body: <p>You can take a look at bayesian hyperparameter optimization as a general method of optimizing loss (or anything) as a function of the hyperparameters. But note that in general the deeper your network the better, so optimizing loss as a function of number of layers isn't a very fun thing to do.</p>
  
  <p>Grid search and a bit of common sense (as learnt by seeing many examples) should be your best bet.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-20T02:11:25.050Z
Answers.LastActivityDate: 2017-10-20T02:11:25.050Z
Answers.UserDisplayName: k.c. sayz 'k.c sayz'
Answers.Body: <p>There is a technique called <code>Pruning</code> in neural networks, which is used just for this same purpose.</p>
  
  <p>The pruning is done on the number of hidden layers. The process is very similar to the pruning process of decision trees. The pruning process is done as follows:</p>
  
  <ul>
  <li>Train a large, densely connected, network with a standard training
  algorithm</li>
  <li>Examine the trained network to assess the relative importance of the
  weights</li>
  <li>Remove the least important weight(s)</li>
  <li>retrain the pruned network</li>
  <li>Repeat steps 2-4 until satisfied</li>
  </ul>
  
  <p>However, there are <a href="https://arxiv.org/abs/1510.00149">several optimized methods</a> for pruning neural nets, and it is also a <a href="http://www.idiap.ch/ftp/reports/1997/rr97-03.pdf">very active area of research</a>.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-20T02:02:57.087Z
Answers.Text: This answer is **incorrect**. Pruning is a method applied onto the weights, not a method to determine the number of layers (a hyperparameter).
Answers.UserDisplayName: k.c. sayz 'k.c sayz'
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-03T06:41:18.960Z
Answers.Text: A symmetric approach is the common "grid search" applied to the network architecture. Start small (so fast), and automatically try larger architectures. All this is just brute force, though, so it is expensive. Yet, when we can start small, the first stages can go pretty fast, and give a better idea of what to aim for.
Answers.UserDisplayName: Eric Platon
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-03T05:35:48.807Z
Answers.LastActivityDate: 2016-08-03T05:35:48.807Z
Answers.UserDisplayName: Dawny33
Body: <p>How do I decide the optimal number of layers for a neural network (feedforward or recurrent)?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-12T18:55:41.530Z
Comments.Text: See also the question [How to choose the number of hidden layers and nodes in a feedforward neural network?](https://stats.stackexchange.com/q/181/82135) at CrossValidated SE.
Comments.UserDisplayName: nbro
Tags: <neural-networks><recurrent-neural-networks><feedforward-neural-networks><hyperparameter-optimization><hidden-layers>
Title: How do I decide the optimal number of layers for a neural network?
UserDisplayName: v01d
ThreadText: # How do I decide the optimal number of layers for a neural network?
  
  (Asked by: username_0 on 2016-08-03)
  
  How do I decide the optimal number of layers for a neural network (feedforward or recurrent)?
  
  **Tags:** neural-networks, recurrent-neural-networks, feedforward-neural-networks, hyperparameter-optimization, hidden-layers
  
  > **Comments:**
  > **username_1**: See also the question [How to choose the number of hidden layers and nodes in a feedforward neural network?](https://stats.stackexchange.com/q/181/82135) at CrossValidated SE.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2016-08-03. Score: 7 | Accepted answer
  
  There is a technique called `Pruning` in neural networks, which is used just for this same purpose.
  
  The pruning is done on the number of hidden layers. The process is very similar to the pruning process of decision trees. The pruning process is done as follows:
  
  * Train a large, densely connected, network with a standard training
    algorithm
  * Examine the trained network to assess the relative importance of the
    weights
  * Remove the least important weight(s)
  * retrain the pruned network
  * Repeat steps 2-4 until satisfied
  
  However, there are [several optimized methods](https://arxiv.org/abs/1510.00149) for pruning neural nets, and it is also a [very active area of research](http://www.idiap.ch/ftp/reports/1997/rr97-03.pdf).
  
  > **Comments:**
  > **username_4**: A symmetric approach is the common "grid search" applied to the network architecture. Start small (so fast), and automatically try larger architectures. All this is just brute force, though, so it is expensive. Yet, when we can start small, the first stages can go pretty fast, and give a better idea of what to aim for.  
  > **username_2**: This answer is **incorrect**. Pruning is a method applied onto the weights, not a method to determine the number of layers (a hyperparameter).  
  
  ---
  
  ### Answer by username_2 on 2017-10-20. Score: 2
  
  You can take a look at bayesian hyperparameter optimization as a general method of optimizing loss (or anything) as a function of the hyperparameters. But note that in general the deeper your network the better, so optimizing loss as a function of number of layers isn't a very fun thing to do.
  
  Grid search and a bit of common sense (as learnt by seeing many examples) should be your best bet.
  
  
  ---
  
  