Answers.Body: <p>There are so many aspects.</p>
  
  <p><strong>1. Training:</strong>
  Training deep nets is a hard job due to the <a href="http://neuralnetworksanddeeplearning.com/chap5.html#the_vanishing_gradient_problem" rel="nofollow noreferrer">vanishing</a> (rearly exploding) gradient problem. So building a 10x100 neural-net is not recommended.</p>
  
  <p><strong>2. Trained network performance:</strong></p>
  
  <ul>
  <li><strong>Information loss:</strong>
  The classical usage of neural nets is the <a href="https://math.stackexchange.com/questions/141381/regression-vs-classification">classification</a> problem. Which means we want to get some well defined information from the data. (Ex. Is there a face in the picture or not.)
  So usually classification problem has a lot of input, and few output, whats more the size of the hidden layers are descend from input to output.
  However, we loss information using less neurons layer by layer. (Ie. We cannot reproduce the original image based on the fact that is there a face on it or no.) So you must know that you loss information using 100 neurons if the size of the input is (lets say) 1000.</li>
  <li><strong>Information complexity:</strong> However the deeper nets (as Tomas W mentioned) can fetch more complex information from the input data. Inspite of this its not recommended to use 10 fully connected layers. Its recommended to use convolutional/relu/maxpooling or other type of layers. Firest layers can compress the some essential part of the inputs. (Ex is there any line in a specific part of the picture) Second layers can say: There is a specific shape in this place in the picture. Etc etc.</li>
  </ul>
  
  <p><strong>So deeper nets are more "clever" but 10x100 net structure is a good choice.</strong></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-05T14:21:31.097Z
Answers.LastActivityDate: 2017-05-05T14:21:31.097Z
Answers.UserDisplayName: betontalpfa
Answers.Body: <p>If the problem you are solving is linearly separable, one layer of 1000 neurons can do better job than 10 layers with each of 100 neurons.
  If the problem is non linear and not convex, then you need deep neural nets. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-24T14:55:53.743Z
Answers.Text: If the problem is linearly separable you don't need hidden layers at all
Answers.UserDisplayName: Amedeo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-17T18:41:57.850Z
Answers.LastActivityDate: 2018-09-17T18:41:57.850Z
Answers.UserDisplayName: Sreedhar Veluri
Answers.Body: <blockquote>
    <p>I think you have a confusion in the basics of the neural networks.
    Every layer has a separate activation function and input/output
    connection weights.</p>
  </blockquote>
  
  <p>The output of the first hidden layer will be multiplied by a weight, processed by an activation function in the next layer and so on.
  Single layer neural networks are very limited for simple tasks, deeper NN can perform far better than a single layer. </p>
  
  <p>However, do not use more than layer if your application is not fairly complex. In conclusion, 100 neurons layer does not mean better neural network than 10 layers x 10 neurons but 10 layers are something imaginary unless you are doing deep learning. start with 10 neurons in the hidden layer and try to add layers or add more neurons to the same layer to see the difference. learning with more layers will be easier but more training time is required.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-05T00:34:03.793Z
Answers.LastActivityDate: 2017-05-06T10:05:20.410Z
Answers.LastEditDate: 2017-05-06T10:05:20.410Z
Answers.UserDisplayName: ahmadjordan
Answers.Body: <p>Basically, having multiple layers (aka a deep network) makes your network more eager to recognize certain aspects of input data. For example, if you have the details of a house (size, lawn size, location etc.) as input and want to predict the price. The first layer may predict:</p>
  
  <ul>
  <li>Big area, higher price</li>
  <li>Small amount of bedrooms, lower price</li>
  </ul>
  
  <p>The second layer might conclude:</p>
  
  <ul>
  <li>Big area + small amount of bedrooms = large bedrooms = +- effect</li>
  </ul>
  
  <p>Yes, one layer can also 'detect' the stats, however it will require more neurons as it cannot rely on other neurons to do 'parts' of the total calculation required to detect that stat.</p>
  
  <p><a href="https://stats.stackexchange.com/questions/274569/deep-networks-vs-shallow-networks-why-do-we-need-depth/274571#274571">Check out this answer</a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-04T19:31:53.020Z
Answers.Text: Thank you so much for your answer. Just to clarify, when you write "it makes your network [...]", are you referring to the case when I have many hidden layers with fewer neurons each rather than having more neurons in fewer layers?
Answers.UserDisplayName: Stephen Johnson
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-04T19:36:36.390Z
Answers.Text: @StephenJohnson oops, I edited the question. I'm referring to the deep network (multiple layers).
Answers.UserDisplayName: Thomas Wagenaar
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-04T20:18:55.833Z
Answers.Text: Nice answer, thanks again. Maybe I should continue this in another thread, but do you think the same kind of reasoning applies to recurrent neural networks such as GRU or LSTM?
Answers.UserDisplayName: Stephen Johnson
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-04T21:20:21.493Z
Answers.Text: @StephenJohnson do you mean one layered recurrent networks vs multilayered recurrent networks or do you mean because of their recurrent connections?
Answers.UserDisplayName: Thomas Wagenaar
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-04T22:22:32.470Z
Answers.Text: I mean generally, because of they have recurrent connections allowing them to map contexts over longer distances, do such networks benefit from being Deep in the same way a regular feedforward network would? Maybe they can't be compared like that since recurrent networks typically are used when sequential data, such as audio, is used.
Answers.UserDisplayName: Stephen Johnson
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-05T08:33:14.443Z
Answers.Text: @StephenJohnson i'm not sure. I just googled "Deep RNN's" and nothing interesting popped up. A reason for this is that RNN's tend to come with predefined architectures (LSTM, GRU, NARX, etc.). And exactly as you say: they are focused on sequential data. But there is one thing: some LSTM's for example have one layer of 5 blocks, and another layer of 5 blocks. So basically deep RNN's can find abstract relationships between **memory blocks** better than single layered RNN's :)
Answers.UserDisplayName: Thomas Wagenaar
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-05T10:04:38.057Z
Answers.Text: Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/58243/discussion-between-stephen-johnson-and-thomas-w).
Answers.UserDisplayName: Stephen Johnson
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-16T00:31:59.947Z
Answers.Text: AFAIK in DNN, the interesting fact is that a hierachy of the knowldge is built throw layers. Bottom layer learn basic concepts and upper one higher concepts. With one layer of 1K neurons, you will mainly construct a kind of loop-up table. With a DNN hierarchy of concepts with is much powerful than the previous NN. But his rely on the assumption that the world is like that: high level concepts are build upon basic one.
Answers.UserDisplayName: jcm69
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-04T13:10:53.287Z
Answers.LastActivityDate: 2017-05-04T19:35:36.130Z
Answers.LastEditDate: 2017-05-04T19:35:36.130Z
Answers.UserDisplayName: Thomas Wagenaar
Body: <p>These types of questions may be problem-dependent, but I have tried to find research that addresses the question whether the number of hidden layers and their size (number of neurons in each layer) really matter or not.</p>
  
  <p>So my question is, does it really matter if we for example have 1 large hidden layer of 1000 neurons vs. 10 hidden layers with 100 neurons each?</p>
  
Tags: <neural-networks>
Title: 1 hidden layer with 1000 neurons vs. 10 hidden layers with 100 neurons
UserDisplayName: Stephen Johnson
ThreadText: # 1 hidden layer with 1000 neurons vs. 10 hidden layers with 100 neurons
  
  (Asked by: username_0 on 2017-05-04)
  
  These types of questions may be problem-dependent, but I have tried to find research that addresses the question whether the number of hidden layers and their size (number of neurons in each layer) really matter or not.
  
  So my question is, does it really matter if we for example have 1 large hidden layer of 1000 neurons vs. 10 hidden layers with 100 neurons each?
  
  **Tags:** neural-networks
  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_5 on 2017-05-04. Score: 14 | Accepted answer
  
  Basically, having multiple layers (aka a deep network) makes your network more eager to recognize certain aspects of input data. For example, if you have the details of a house (size, lawn size, location etc.) as input and want to predict the price. The first layer may predict:
  
  * Big area, higher price
  * Small amount of bedrooms, lower price
  
  The second layer might conclude:
  
  * Big area + small amount of bedrooms = large bedrooms = +- effect
  
  Yes, one layer can also 'detect' the stats, however it will require more neurons as it cannot rely on other neurons to do 'parts' of the total calculation required to detect that stat.
  
  [Check out this answer](https://stats.stackexchange.com/questions/274569/deep-networks-vs-shallow-networks-why-do-we-need-depth/274571#274571)
  
  > **Comments:**
  > **username_0**: Thank you so much for your answer. Just to clarify, when you write "it makes your network [...]", are you referring to the case when I have many hidden layers with fewer neurons each rather than having more neurons in fewer layers?  
  > **username_5**: @username_0 oops, I edited the question. I'm referring to the deep network (multiple layers).  
  > **username_0**: Nice answer, thanks again. Maybe I should continue this in another thread, but do you think the same kind of reasoning applies to recurrent neural networks such as GRU or LSTM?  
  > **username_5**: @username_0 do you mean one layered recurrent networks vs multilayered recurrent networks or do you mean because of their recurrent connections?  
  > **username_0**: I mean generally, because of they have recurrent connections allowing them to map contexts over longer distances, do such networks benefit from being Deep in the same way a regular feedforward network would? Maybe they can't be compared like that since recurrent networks typically are used when sequential data, such as audio, is used.  
  > **username_5**: @username_0 i'm not sure. I just googled "Deep RNN's" and nothing interesting popped up. A reason for this is that RNN's tend to come with predefined architectures (LSTM, GRU, NARX, etc.). And exactly as you say: they are focused on sequential data. But there is one thing: some LSTM's for example have one layer of 5 blocks, and another layer of 5 blocks. So basically deep RNN's can find abstract relationships between **memory blocks** better than single layered RNN's :)  
  > **username_0**: Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/58243/discussion-between-username_0-username_0-and-username_5-w).  
  > **username_6**: AFAIK in DNN, the interesting fact is that a hierachy of the knowldge is built throw layers. Bottom layer learn basic concepts and upper one higher concepts. With one layer of 1K neurons, you will mainly construct a kind of loop-up table. With a DNN hierarchy of concepts with is much powerful than the previous NN. But his rely on the assumption that the world is like that: high level concepts are build upon basic one.  
  
  ---
  
  ### Answer by username_1 on 2017-05-05. Score: 4
  
  There are so many aspects.
  
  **1. Training:**
  Training deep nets is a hard job due to the [vanishing](http://neuralnetworksanddeeplearning.com/chap5.html#the_vanishing_gradient_problem) (rearly exploding) gradient problem. So building a 10x100 neural-net is not recommended.
  
  **2. Trained network performance:**
  
  * **Information loss:**
    The classical usage of neural nets is the [classification](https://math.stackexchange.com/questions/141381/regression-vs-classification) problem. Which means we want to get some well defined information from the data. (Ex. Is there a face in the picture or not.)
    So usually classification problem has a lot of input, and few output, whats more the size of the hidden layers are descend from input to output.
    However, we loss information using less neurons layer by layer. (Ie. We cannot reproduce the original image based on the fact that is there a face on it or no.) So you must know that you loss information using 100 neurons if the size of the input is (lets say) 1000.
  * **Information complexity:** However the deeper nets (as Tomas W mentioned) can fetch more complex information from the input data. Inspite of this its not recommended to use 10 fully connected layers. Its recommended to use convolutional/relu/maxpooling or other type of layers. Firest layers can compress the some essential part of the inputs. (Ex is there any line in a specific part of the picture) Second layers can say: There is a specific shape in this place in the picture. Etc etc.
  
  **So deeper nets are more "clever" but 10x100 net structure is a good choice.**
  
  
  ---
  
  ### Answer by username_2 on 2018-09-17. Score: 1
  
  If the problem you are solving is linearly separable, one layer of 1000 neurons can do better job than 10 layers with each of 100 neurons.
  If the problem is non linear and not convex, then you need deep neural nets.
  
  > **Comments:**
  > **username_3**: If the problem is linearly separable you don't need hidden layers at all  
  
  ---
  
  ### Answer by username_4 on 2017-05-05. Score: 0
  
  > I think you have a confusion in the basics of the neural networks.
  > Every layer has a separate activation function and input/output
  > connection weights.
  
  The output of the first hidden layer will be multiplied by a weight, processed by an activation function in the next layer and so on.
  Single layer neural networks are very limited for simple tasks, deeper NN can perform far better than a single layer.
  
  However, do not use more than layer if your application is not fairly complex. In conclusion, 100 neurons layer does not mean better neural network than 10 layers x 10 neurons but 10 layers are something imaginary unless you are doing deep learning. start with 10 neurons in the hidden layer and try to add layers or add more neurons to the same layer to see the difference. learning with more layers will be easier but more training time is required.
  
  
  ---
  
  