Answers.Body: <p>Paper <a href="https://arxiv.org/pdf/1512.00567.pdf" rel="noreferrer">Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the inception architecture for computer vision[J]. arXiv preprint arXiv:1512.00567, 2015.</a> gives some general design principles:</p>
  
  <blockquote>
    <ol>
    <li><p>Avoid representational bottlenecks, especially early in
    the network;</p></li>
    <li><p>Balance the width and depth of the network. Optimal
    performance of the network can be reached by balancing
    the number of filters per stage and the depth of
    the network. Increasing both the width and the depth
    of the network can contribute to higher quality networks.
    However, the optimal improvement for a constant
    amount of computation can be reached if both are
    increased in parallel. The computational budget should
    therefore be distributed in a balanced way between the
    depth and width of the network.</p></li>
    </ol>
  </blockquote>
  
  <p>These suggestions can't bring you the optimal number of neurons in a network though.</p>
  
  <p>However, there are still some model compression research e.g. <a href="https://github.com/wenwei202/caffe/tree/scnn" rel="noreferrer">Structured Sparsity Learning (SSL) of Deep Neural Networks</a>, <a href="https://github.com/songhan/SqueezeNet-Deep-Compression" rel="noreferrer">SqueezeNet</a>, <a href="http://papers.nips.cc/paper/5784-learning-both-weights-and-connections-for-efficient-neural-network.pdf" rel="noreferrer">Pruning network</a> that may shed some light on how to optimizing the neurons per single layer.</p>
  
  <p>Especially in <a href="https://github.com/wenwei202/caffe/tree/scnn" rel="noreferrer">Structured Sparsity Learning of Deep Neural Networks</a>, it adds a <code>Group Lasso</code> regularization term in the loss function to to regularize the structures(i.e., filters, channels, filter shapes, and layer depth) of DNNs, which namely is to zero out some components(i.e., filters, channels, filter shapes, and layer depth) of the net structure and achieves a remarkable compact and acceleration of the network, while keeps a small classification accuracy loss. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-10-06T02:57:54.343Z
Answers.LastActivityDate: 2016-10-11T01:02:00.430Z
Answers.LastEditDate: 2016-10-11T01:02:00.430Z
Answers.UserDisplayName: Dale
Answers.Body: <p>For a more intelligent approach than random or exhaustive searches, you could try a genetic algorithm such as NEAT <a href="http://nn.cs.utexas.edu/?neat">http://nn.cs.utexas.edu/?neat</a>. However, this has no guarantee to find a global optima, it is simply an optimization algorithm based on performance and is therefore vulnerable to getting stuck in a local optima. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-11T14:50:04.153Z
Answers.LastActivityDate: 2016-08-11T14:50:04.153Z
Answers.UserDisplayName: Tim Atkinson
Answers.Body: <p>There is no direct way to find the optimal number of them: people empirically try and see (e.g., using cross-validation). The most common search techniques are random, manual, and grid searches. </p>
  
  <p>There exist more advanced techniques such as Gaussian processes, e.g. <em><a href="http://arxiv.org/abs/1609.08703" rel="noreferrer">Optimizing Neural Network Hyperparameters with Gaussian Processes for Dialog Act Classification</a>, IEEE SLT 2016</em>.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-02T15:50:27.867Z
Answers.LastActivityDate: 2016-09-29T00:24:06.177Z
Answers.LastEditDate: 2016-09-29T00:24:06.177Z
Answers.UserDisplayName: Franck Dernoncourt
Body: <p>When you're writing your algorithm, how do you know how many neurons you need per single layer? Are there any methods for finding the optimal number of them, or is it a rule of thumb?</p>
  
Tags: <neural-networks><hyperparameter-optimization><artificial-neuron><hyper-parameters><layers>
Title: How to find the optimal number of neurons per layer?
UserDisplayName: kenorb
ThreadText: # How to find the optimal number of neurons per layer?
  
  (Asked by: username_0 on 2016-08-02)
  
  When you're writing your algorithm, how do you know how many neurons you need per single layer? Are there any methods for finding the optimal number of them, or is it a rule of thumb?
  
  **Tags:** neural-networks, hyperparameter-optimization, artificial-neuron, hyper-parameters, layers
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2016-08-02. Score: 19 | Accepted answer
  
  There is no direct way to find the optimal number of them: people empirically try and see (e.g., using cross-validation). The most common search techniques are random, manual, and grid searches.
  
  There exist more advanced techniques such as Gaussian processes, e.g. *[Optimizing Neural Network Hyperparameters with Gaussian Processes for Dialog Act Classification](http://arxiv.org/abs/1609.08703), IEEE SLT 2016*.
  
  
  ---
  
  ### Answer by username_2 on 2016-08-11. Score: 7
  
  For a more intelligent approach than random or exhaustive searches, you could try a genetic algorithm such as NEAT <http://nn.cs.utexas.edu/?neat>. However, this has no guarantee to find a global optima, it is simply an optimization algorithm based on performance and is therefore vulnerable to getting stuck in a local optima.
  
  
  ---
  
  ### Answer by username_1 on 2016-10-06. Score: 6
  
  Paper [Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the inception architecture for computer vision[J]. arXiv preprint arXiv:1512.00567, 2015.](https://arxiv.org/pdf/1512.00567.pdf) gives some general design principles:
  
  > 1. Avoid representational bottlenecks, especially early in
  >    the network;
  > 2. Balance the width and depth of the network. Optimal
  >    performance of the network can be reached by balancing
  >    the number of filters per stage and the depth of
  >    the network. Increasing both the width and the depth
  >    of the network can contribute to higher quality networks.
  >    However, the optimal improvement for a constant
  >    amount of computation can be reached if both are
  >    increased in parallel. The computational budget should
  >    therefore be distributed in a balanced way between the
  >    depth and width of the network.
  
  These suggestions can't bring you the optimal number of neurons in a network though.
  
  However, there are still some model compression research e.g. [Structured Sparsity Learning (SSL) of Deep Neural Networks](https://github.com/wenwei202/caffe/tree/scnn), [SqueezeNet](https://github.com/songhan/SqueezeNet-Deep-Compression), [Pruning network](http://papers.nips.cc/paper/5784-learning-both-weights-and-connections-for-efficient-neural-network.pdf) that may shed some light on how to optimizing the neurons per single layer.
  
  Especially in [Structured Sparsity Learning of Deep Neural Networks](https://github.com/wenwei202/caffe/tree/scnn), it adds a `Group Lasso` regularization term in the loss function to to regularize the structures(i.e., filters, channels, filter shapes, and layer depth) of DNNs, which namely is to zero out some components(i.e., filters, channels, filter shapes, and layer depth) of the net structure and achieves a remarkable compact and acceleration of the network, while keeps a small classification accuracy loss.
  
  
  ---
  
  