Answers.Body: <p>A very wide but shallow neural network is going to be harder to train.<br />
  You can check that with the playground of tensorflow or with the MPG example in Google Colab.<br />
  The relationship between architecture and learning capabilities is not fully understood, but, empirically, thats what you see.<br />
  But making the network too deep creates more problems:</p>
  <ul>
  <li>Vanishing gradients.</li>
  <li>Less parameters for the same amount of neurons.</li>
  <li>Exploding gradients.</li>
  </ul>
  <p>It is for that reason that humans and neural networks are the one deciding a good architecture.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-09T14:52:45.843Z
Answers.Text: "understud" => understood. Also, vanishing gradients typically occur as you increase the depth and not width?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-08T21:42:09.780Z
Answers.LastActivityDate: 2020-10-10T20:35:42.657Z
Answers.LastEditDate: 2020-10-10T20:35:42.657Z
Answers.UserDisplayName: Joaquin Torrens
Body: <p>I know that my question probably seems like being asked many times, but Ill try
  to be more speciffic:</p>
  <p><strong>Limitations to my question:</strong></p>
  <ol>
  <li><p>I am NOT asking about convolutional neural networks, so please, try not to mention this as an example or as an answer as long as it is possible. (maybe only in question number 3)</p>
  </li>
  <li><p>My question is NOT about classification using neural networks</p>
  </li>
  <li><p>I am asking about a &quot;simple&quot; neural network designed to solve the regression type of problem. Let's say it has 2 inputs and 1 output.</p>
  </li>
  </ol>
  <p><strong>Preambula:</strong></p>
  <p>As far as I understood, from the <em>universal approximation theorem</em>, in such a case, even if the model is nonlinear, only one hidden layer can perfectly fit a nonlinear model, as shown here
  <a href="http://neuralnetworksanddeeplearning.com/chap4.html" rel="nofollow noreferrer">http://neuralnetworksanddeeplearning.com/chap4.html</a>.</p>
  <p><strong>Question 1</strong></p>
  <p>In this specific case, is there any added value in using extra layers?
  (maybe the model will be more precise, or faster training?)</p>
  <p><strong>Question 2</strong></p>
  <p>Suppose in 1st question the answer was there is no added value. In such a case will the added value appear if I enlarge inputs from two inputs as described above, to some larger number?</p>
  <p><strong>Question 3</strong></p>
  <p>Suppose in 2nd question the answer was there is no added value. I am still trying to pinpoint the situation where it STARTS making sense in adding more layers AND where it makes NO sense at all using one layer.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-09T08:36:50.870Z
Comments.Text: Please, ask one question per post. If you have multiple questions, ask each of them in its separate post. Take a look at https://ai.stackexchange.com/help/on-topic for more info about our site.
Comments.UserDisplayName: nbro
Tags: <neural-networks><regression><hidden-layers><non-linear-regression><universal-approximation-theorems>
Title: When are multiple hidden layers necessary?
UserDisplayName: Igor
ThreadText: # When are multiple hidden layers necessary?
  
  (Asked by: username_0 on 2020-10-08)
  
  I know that my question probably seems like being asked many times, but Ill try
  to be more speciffic:
  
  **Limitations to my question:**
  
  1. I am NOT asking about convolutional neural networks, so please, try not to mention this as an example or as an answer as long as it is possible. (maybe only in question number 3)
  2. My question is NOT about classification using neural networks
  3. I am asking about a "simple" neural network designed to solve the regression type of problem. Let's say it has 2 inputs and 1 output.
  
  **Preambula:**
  
  As far as I understood, from the *universal approximation theorem*, in such a case, even if the model is nonlinear, only one hidden layer can perfectly fit a nonlinear model, as shown here
  <http://neuralnetworksanddeeplearning.com/chap4.html>.
  
  **Question 1**
  
  In this specific case, is there any added value in using extra layers?
  (maybe the model will be more precise, or faster training?)
  
  **Question 2**
  
  Suppose in 1st question the answer was there is no added value. In such a case will the added value appear if I enlarge inputs from two inputs as described above, to some larger number?
  
  **Question 3**
  
  Suppose in 2nd question the answer was there is no added value. I am still trying to pinpoint the situation where it STARTS making sense in adding more layers AND where it makes NO sense at all using one layer.
  
  **Tags:** neural-networks, regression, hidden-layers, non-linear-regression, universal-approximation-theorems
  > **Comments:**
  > **username_1**: Please, ask one question per post. If you have multiple questions, ask each of them in its separate post. Take a look at https://ai.stackexchange.com/help/on-topic for more info about our site.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-10-08. Score: 1 | Accepted answer
  
  A very wide but shallow neural network is going to be harder to train.  
  You can check that with the playground of tensorflow or with the MPG example in Google Colab.  
  The relationship between architecture and learning capabilities is not fully understood, but, empirically, thats what you see.  
  But making the network too deep creates more problems:
  
  * Vanishing gradients.
  * Less parameters for the same amount of neurons.
  * Exploding gradients.
  
  It is for that reason that humans and neural networks are the one deciding a good architecture.
  
  > **Comments:**
  > **username_1**: "understud" => understood. Also, vanishing gradients typically occur as you increase the depth and not width?  
  
  ---
  
  