Answers.Body: <p>A deeper network will have more capacity, regardless of whether new layers have the same number of neurons as the previous layers, fewer, or more.</p>
  <p>More capacity means that in theory the network can learn more complex functions. It does not mean that the network will then be better on any given task.</p>
  <p>One important thing to note: This is only the case if you have a non linear activation function. If you do not have a non linear activation function between layers, then you are correct that each added layer, despite adding more parameters, does not add capacity to the network.</p>
  <p>For further reference, I recommend reading this chapter of the deep learning book: <a href="https://www.deeplearningbook.org/contents/mlp.html" rel="nofollow noreferrer">https://www.deeplearningbook.org/contents/mlp.html</a></p>
  <p>which contains all the information above (perhaps with more technical jargon) and more.</p>
  <p>For a greater understanding of how adding layers affects capacity,  this article will be useful: <a href="https://en.wikipedia.org/wiki/Universal_approximation_theorem" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Universal_approximation_theorem</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-09T03:23:29.330Z
Answers.Text: "If you do not have a non linear activation function" - is that simply saying you have a linear activation function? 
Answers.UserDisplayName: eddiewould
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-09T04:23:55.003Z
Answers.Text: @eddiewould Nice reversal of the double negative :) But it also could mean that you lack an activation function entirely.
Answers.UserDisplayName: chessprogrammer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-01T07:12:22.240Z
Answers.LastActivityDate: 2022-12-01T07:12:22.240Z
Answers.UserDisplayName: chessprogrammer
Body: <p>Lets say I have a neural network with three layers and the last layer has 3 outputs.
  If I added additional layer of 3 neurons to the end of the network, would that be a more powerful neural network?</p>
  <p>Here's an example picture of what I mean:
  <a href="https://i.sstatic.net/dFO55l.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/dFO55l.png" alt="enter image description here" /></a></p>
  <p>(where the circles are the neurons in the network, L1 is layer1. Also assume that all of the neurons are fully-connected to the neurons of the previous layer.)</p>
  <p>Typically neural networks that I see the amount of neurons per network typically decrease as there are layers. But is it theoretically possible to a really powerful network by signiciantly increasing the depth of the network?</p>
  <p>For example consider the following case:</p>
  <p><a href="https://i.sstatic.net/ShbgZ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ShbgZ.png" alt="enter image description here" /></a>
  Does stacking extra layers on the end of a neural network allow it to learn more complex tasks? My guess would be that stacking extra layers only helps if you are distilling the information down to less neurons per layer. But with more advanced neural networks like GPT-3, what has been said to me is what is important is the total amount of tunable parameters, and that the particular structure or depth of the network isn't important (when holding the total number of tunable parameters constant).</p>
  <p>So phrasing this another way: If, for example, I wanted to create a neural network at the scale of a GPT-3 level, (ignoring the intricacies of the transformer architecture and just assuming that it is a basic neural network), is it possible that instead of adding additional neurons per layer, I can simply add more layers of the same neuron size to scale-up the total weight parameters?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-23T02:54:40.883Z
Comments.Text: Think again. Where does "the total amount of tunable parameters" come from? From "the amount of nodes or layers". It does not matter how big or small the network is.
Comments.UserDisplayName: lpounng
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-23T14:40:40.090Z
Comments.Text: @lpounng, sorry but I dont understand what you mean here. Of course a larger network (one with more parameters) is neccessary for more complicated problems -- so I don't follow.
Comments.UserDisplayName: Steven Sagona
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-28T08:17:14.467Z
Comments.Text: Referring to this line, "...what is important is the total amount of tunable parameters, and that the amount of nodes or layers is not as important".
Comments.UserDisplayName: lpounng
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-29T15:25:25.330Z
Comments.Text: I know what line you are referring to, but don't understand your question.
Comments.UserDisplayName: Steven Sagona
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-30T01:23:35.497Z
Comments.Text: It was not a question, but a statement. "the total amount of tunable parameters" is proportional to "the amount of nodes or layers", regardless of the scale of model. So the statement "what is important is the total amount of tunable parameters, and that the amount of nodes or layers is not as important" does not make sense.
Comments.UserDisplayName: lpounng
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-30T15:56:27.417Z
Comments.Text: I edited that sentence to make it more clear.
Comments.UserDisplayName: Steven Sagona
Tags: <neural-networks><deep-learning><deep-neural-networks>
Title: Is it redunant to add more layers to a neural network with same number of neurons as the previous layer?
UserDisplayName: Steven Sagona
ThreadText: # Is it redunant to add more layers to a neural network with same number of neurons as the previous layer?
  
  (Asked by: username_0 on 2022-11-22)
  
  Lets say I have a neural network with three layers and the last layer has 3 outputs.
  If I added additional layer of 3 neurons to the end of the network, would that be a more powerful neural network?
  
  Here's an example picture of what I mean:
  [![enter image description here](https://i.sstatic.net/dFO55l.png)](https://i.sstatic.net/dFO55l.png)
  
  (where the circles are the neurons in the network, L1 is layer1. Also assume that all of the neurons are fully-connected to the neurons of the previous layer.)
  
  Typically neural networks that I see the amount of neurons per network typically decrease as there are layers. But is it theoretically possible to a really powerful network by signiciantly increasing the depth of the network?
  
  For example consider the following case:
  
  [![enter image description here](https://i.sstatic.net/ShbgZ.png)](https://i.sstatic.net/ShbgZ.png)
  Does stacking extra layers on the end of a neural network allow it to learn more complex tasks? My guess would be that stacking extra layers only helps if you are distilling the information down to less neurons per layer. But with more advanced neural networks like GPT-3, what has been said to me is what is important is the total amount of tunable parameters, and that the particular structure or depth of the network isn't important (when holding the total number of tunable parameters constant).
  
  So phrasing this another way: If, for example, I wanted to create a neural network at the scale of a GPT-3 level, (ignoring the intricacies of the transformer architecture and just assuming that it is a basic neural network), is it possible that instead of adding additional neurons per layer, I can simply add more layers of the same neuron size to scale-up the total weight parameters?
  
  **Tags:** neural-networks, deep-learning, deep-neural-networks
  > **Comments:**
  > **username_1**: Think again. Where does "the total amount of tunable parameters" come from? From "the amount of nodes or layers". It does not matter how big or small the network is.  
  > **username_0**: @lpounng, sorry but I dont understand what you mean here. Of course a larger network (one with more parameters) is neccessary for more complicated problems -- so I don't follow.  
  > **username_1**: Referring to this line, "...what is important is the total amount of tunable parameters, and that the amount of nodes or layers is not as important".  
  > **username_0**: I know what line you are referring to, but don't understand your question.  
  > **username_1**: It was not a question, but a statement. "the total amount of tunable parameters" is proportional to "the amount of nodes or layers", regardless of the scale of model. So the statement "what is important is the total amount of tunable parameters, and that the amount of nodes or layers is not as important" does not make sense.  
  > **username_0**: I edited that sentence to make it more clear.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-12-01. Score: 2
  
  A deeper network will have more capacity, regardless of whether new layers have the same number of neurons as the previous layers, fewer, or more.
  
  More capacity means that in theory the network can learn more complex functions. It does not mean that the network will then be better on any given task.
  
  One important thing to note: This is only the case if you have a non linear activation function. If you do not have a non linear activation function between layers, then you are correct that each added layer, despite adding more parameters, does not add capacity to the network.
  
  For further reference, I recommend reading this chapter of the deep learning book: <https://www.deeplearningbook.org/contents/mlp.html>
  
  which contains all the information above (perhaps with more technical jargon) and more.
  
  For a greater understanding of how adding layers affects capacity, this article will be useful: <https://en.wikipedia.org/wiki/Universal_approximation_theorem>
  
  > **Comments:**
  > **username_3**: "If you do not have a non linear activation function" - is that simply saying you have a linear activation function?   
  > **username_2**: @eddiewould Nice reversal of the double negative :) But it also could mean that you lack an activation function entirely.  
  
  ---
  
  