Answers.Body: <p>Yes, NEAT (NeuroEvolution of Augmenting Topologies) increases the number of neurons during training. More specifically, NEAT uses evolution to introduce new neurons and connections during training, and - just as evolution - if the mutation performs poorly, gets eliminated after a few generations. This way overall performance increases over time while it keeps your network size (and computing power to run it) minimal. There's also a way to add convolution to this algorithm.</p>
  
  <p>In the <a href="http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf" rel="nofollow noreferrer">paper that introduced NEAT</a>, the authors mention a completely different algorithm they tried, optimizing network hyper-parameters by choosing reasonable random values and training it multiple times. That could decrease that number as well.</p>
  
  <p>Also, there is a trick to temporarily turn off specific parts of a network, which supposedly helps with overfitting.</p>
  
  <p>A ReLU+backpropagation based network can "turn off" parts of the network during training, because constant 0's derivative is constant 0. In practice, you decrease the number of neurons (that is considered bad though, that's when leaky ReLU and PReLU is used instead).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-04T12:42:45.213Z
Answers.Text: NEAT uses evolution to introduce new neurons and connections during training, and - just as evolution - if the mutation performs poorly, gets eliminated after a few generations. This way overall performance increases over time while it keeps your network size (and computing power to run it) minimal. There's also a way to add convolution to this algo.
Answers.UserDisplayName: Nyos
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-04T10:37:41.157Z
Answers.LastActivityDate: 2019-03-04T21:00:40.673Z
Answers.LastEditDate: 2019-03-04T21:00:40.673Z
Answers.UserDisplayName: Nyos
Body: <p>Is there some type of neural network that changes the number of neurons while training? </p>
  
  <p>Using this idea, the network can increase or decrease the number of neurons when the complexity of the inputs increases or decreases.</p>
  
Tags: <neural-networks><neat><neuroevolution>
Title: Is there a neural network with a varying number of neurons?
UserDisplayName: Aura Lee
ThreadText: # Is there a neural network with a varying number of neurons?
  
  (Asked by: username_0 on 2019-03-03)
  
  Is there some type of neural network that changes the number of neurons while training?
  
  Using this idea, the network can increase or decrease the number of neurons when the complexity of the inputs increases or decreases.
  
  **Tags:** neural-networks, neat, neuroevolution
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-04. Score: 4 | Accepted answer
  
  Yes, NEAT (NeuroEvolution of Augmenting Topologies) increases the number of neurons during training. More specifically, NEAT uses evolution to introduce new neurons and connections during training, and - just as evolution - if the mutation performs poorly, gets eliminated after a few generations. This way overall performance increases over time while it keeps your network size (and computing power to run it) minimal. There's also a way to add convolution to this algorithm.
  
  In the [paper that introduced NEAT](http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf), the authors mention a completely different algorithm they tried, optimizing network hyper-parameters by choosing reasonable random values and training it multiple times. That could decrease that number as well.
  
  Also, there is a trick to temporarily turn off specific parts of a network, which supposedly helps with overfitting.
  
  A ReLU+backpropagation based network can "turn off" parts of the network during training, because constant 0's derivative is constant 0. In practice, you decrease the number of neurons (that is considered bad though, that's when leaky ReLU and PReLU is used instead).
  
  > **Comments:**
  > **username_1**: NEAT uses evolution to introduce new neurons and connections during training, and - just as evolution - if the mutation performs poorly, gets eliminated after a few generations. This way overall performance increases over time while it keeps your network size (and computing power to run it) minimal. There's also a way to add convolution to this algo.  
  
  ---
  
  