Answers.Body: <p>More hidden layers will just escalate the possibilities amount the neurons, including the solutions from the previous hidden layers. (I will edit this once I am at home and provide you with a good link I found some time ago)</p>
  
  <p>Meanwhile maybe this will help you <a href="https://stats.stackexchange.com/questions/63152/what-does-the-hidden-layer-in-a-neural-network-compute">https://stats.stackexchange.com/questions/63152/what-does-the-hidden-layer-in-a-neural-network-compute</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-18T15:56:04.677Z
Answers.LastActivityDate: 2019-02-18T15:56:04.677Z
Answers.UserDisplayName: Fleep
Answers.Body: <p>It was proven a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions (see <a href="https://en.wikipedia.org/wiki/Universal_approximation_theorem" rel="nofollow noreferrer">Universal approximation theorem</a>).</p>
  
  <p>More layers can't improve something that can already do "everything". But adding more layers reduces the number of necessary neurons, and reduces computing power needed for the network as well.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-18T22:40:24.617Z
Answers.LastActivityDate: 2019-02-18T22:40:24.617Z
Answers.UserDisplayName: Nyos
Answers.Body: <p>This is a very interesting question that you ask. I believe, this <a href="http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/" rel="nofollow noreferrer">post</a> and this <a href="http://penkovsky.com/post/neural-networks-2/" rel="nofollow noreferrer">post</a> (written by me) well address your question. However, it deserves an explanation here.</p>
  <p><strong>1. Fully connected networks</strong></p>
  <p>The more layers you add, the more &quot;nonlinear&quot; your network becomes. For instance, in the case of <a href="https://www.gwern.net/docs/ai/1988-lang.pdf" rel="nofollow noreferrer">two spirals problem</a>, which requires a &quot;highly nonlinear separation&quot;, the <a href="https://www.gwern.net/docs/ai/1988-lang.pdf" rel="nofollow noreferrer">first known architecture</a> to solve the problem was pretty advanced for that time: it had 3 hidden layers and also it had skip-connections (very early <em>ResNet</em> in 1988). Back then, computing was a way less powerful and training methods with momentum were not known. Nevertheless, thanks to the multilayer architecture, the problem was solved. <a href="http://penkovsky.com/post/neural-networks-2/" rel="nofollow noreferrer">Here</a>, however, I was able to train a single-hidden layer network to solve the spirals problem using Adam.</p>
  <p><strong>2. Convolutional nets (CNNs)</strong></p>
  <p>An interesting partial case of neural networks are CNNs. They restrict the architecture of the first layers, known as <em>convolutional</em> layers, so that there is a much smaller number of trainable parameters due to the weights sharing. What we have learned from computer vision, moving towards the end of CNNs layers, their receptive fields become larger. That means that the subsequent CNN layers &quot;see&quot; more than their predecessors. Conceptually, first CNN layers can recognize simpler features such as edges and textures, whereas final CNN layers contain information about more abstract objects such as trees or faces.</p>
  <p><strong>3. Recurrent nets (RNNs)</strong></p>
  <p>RNNs are networks with layers which receive some of their outputs as inputs. Technically, a single recurrent layer is equivalent to an infinite (or at least large) number of ordinary layers. Thanks to that recurrence, RNNs retain an internal state (memory). Therefore, it is much more difficult to answer your question in the case of recurrent nets. What is known, due to their memory, RNNs are more like programs, and thus are in principle more complex than other neural networks. Please let me know if you find an answer to your question in the last case.</p>
  <p>To conclude, the higher number of hidden layers may help to structure a neural network. Thanks to the recent developments such as <a href="https://arxiv.org/abs/1512.03385" rel="nofollow noreferrer">ResNets</a> and <a href="https://en.wikipedia.org/wiki/Backpropagation_through_time" rel="nofollow noreferrer">backpropagation through time</a>, it is possible to train neural networks with a large number of hidden layers.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-21T17:32:54.910Z
Answers.LastActivityDate: 2022-05-25T09:02:45.307Z
Answers.LastEditDate: 2022-05-25T09:02:45.307Z
Answers.UserDisplayName: penkovsky
Body: <p>In a neural network, the number of neurons in the hidden layer corresponds to the complexity of the model generated to map the inputs to output(s).  More neurons creates a more complex function (and thus the ability to model more nuanced decision barriers) than a hidden layer with less nodes.</p>
  
  <p>But what of the hidden layers?  What do more hidden layers correspond to in terms of the model generated?</p>
  
Tags: <neural-networks><models><hidden-layers>
Title: To what does the number of hidden layers in a neural network correspond?
UserDisplayName: SeeDerekEngineer
ThreadText: # To what does the number of hidden layers in a neural network correspond?
  
  (Asked by: username_0 on 2019-02-18)
  
  In a neural network, the number of neurons in the hidden layer corresponds to the complexity of the model generated to map the inputs to output(s). More neurons creates a more complex function (and thus the ability to model more nuanced decision barriers) than a hidden layer with less nodes.
  
  But what of the hidden layers? What do more hidden layers correspond to in terms of the model generated?
  
  **Tags:** neural-networks, models, hidden-layers
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2019-03-21. Score: 2 | Accepted answer
  
  This is a very interesting question that you ask. I believe, this [post](http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/) and this [post](http://username_3.com/post/neural-networks-2/) (written by me) well address your question. However, it deserves an explanation here.
  
  **1. Fully connected networks**
  
  The more layers you add, the more "nonlinear" your network becomes. For instance, in the case of [two spirals problem](https://www.gwern.net/docs/ai/1988-lang.pdf), which requires a "highly nonlinear separation", the [first known architecture](https://www.gwern.net/docs/ai/1988-lang.pdf) to solve the problem was pretty advanced for that time: it had 3 hidden layers and also it had skip-connections (very early *ResNet* in 1988). Back then, computing was a way less powerful and training methods with momentum were not known. Nevertheless, thanks to the multilayer architecture, the problem was solved. [Here](http://username_3.com/post/neural-networks-2/), however, I was able to train a single-hidden layer network to solve the spirals problem using Adam.
  
  **2. Convolutional nets (CNNs)**
  
  An interesting partial case of neural networks are CNNs. They restrict the architecture of the first layers, known as *convolutional* layers, so that there is a much smaller number of trainable parameters due to the weights sharing. What we have learned from computer vision, moving towards the end of CNNs layers, their receptive fields become larger. That means that the subsequent CNN layers "see" more than their predecessors. Conceptually, first CNN layers can recognize simpler features such as edges and textures, whereas final CNN layers contain information about more abstract objects such as trees or faces.
  
  **3. Recurrent nets (RNNs)**
  
  RNNs are networks with layers which receive some of their outputs as inputs. Technically, a single recurrent layer is equivalent to an infinite (or at least large) number of ordinary layers. Thanks to that recurrence, RNNs retain an internal state (memory). Therefore, it is much more difficult to answer your question in the case of recurrent nets. What is known, due to their memory, RNNs are more like programs, and thus are in principle more complex than other neural networks. Please let me know if you find an answer to your question in the last case.
  
  To conclude, the higher number of hidden layers may help to structure a neural network. Thanks to the recent developments such as [ResNets](https://arxiv.org/abs/1512.03385) and [backpropagation through time](https://en.wikipedia.org/wiki/Backpropagation_through_time), it is possible to train neural networks with a large number of hidden layers.
  
  
  ---
  
  ### Answer by username_1 on 2019-02-18. Score: 1
  
  More hidden layers will just escalate the possibilities amount the neurons, including the solutions from the previous hidden layers. (I will edit this once I am at home and provide you with a good link I found some time ago)
  
  Meanwhile maybe this will help you <https://stats.stackexchange.com/questions/63152/what-does-the-hidden-layer-in-a-neural-network-compute>
  
  
  ---
  
  ### Answer by username_2 on 2019-02-18. Score: 1
  
  It was proven a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions (see [Universal approximation theorem](https://en.wikipedia.org/wiki/Universal_approximation_theorem)).
  
  More layers can't improve something that can already do "everything". But adding more layers reduces the number of necessary neurons, and reduces computing power needed for the network as well.
  
  
  ---
  
  