Answers.Body: <p>There are <strong>many</strong> problems requiring more than two hidden layers. Randomly select a recent Google journal paper on deep learning, you'll see their network could have something like 5 (or more) hidden layers.</p>
  
  <p>Justin Domke wrote his notes for students, so he probably tried to make his points as simple as possible. For a "typical" machine learning problem that students would most likely work on, two hidden layers should be sufficient. But that doesn't add up for a real practical problem. "Deep" learning usually mean more than two hidden layers.</p>
  
  <p>Number of hidden layers is network design that nobody knows for sure. Yoshua Bengio's suggestion is common and simple. It's not a mathmatic proof, but simply a guideline if you don't know what to  do. You just repeat and repeat, until you see the test error no longer improve.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-28T02:37:00.607Z
Answers.LastActivityDate: 2017-08-28T02:42:22.073Z
Answers.LastEditDate: 2017-08-28T02:42:22.073Z
Answers.UserDisplayName: SmallChess
Answers.Body: <p>Your first link is from 2011, which essentially predates the current deep learning explosion. In the many years that have since passed (AlexNet 2012, ResNet 2015) we have since found that if you keep adding layers, we generally do see improved performance.</p>
  
  <p><a href="https://i.sstatic.net/chMZQ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/chMZQ.png" alt="enter image description here"></a></p>
  
  <p>This is due to improved training techniques and optimization breakthroughs (residual connections, ReLU, dropout etc.). But do note that the result can be diminishing. In particular, take a look at <a href="https://arxiv.org/pdf/1909.01377.pdf" rel="nofollow noreferrer">Deep Equillibrium Models</a>, which essentially allow us to train (in the limit equivalence) <em>infinite depth</em> neural networks. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-12T19:48:58.233Z
Answers.LastActivityDate: 2020-04-12T19:48:58.233Z
Answers.UserDisplayName: information_interchange
Answers.Body: <p>In fact they are telling the same thing: Plot the x-axis as the number of hidden layers, and the y-axis as the performance (e.g. classification accuracy), then this curve will have an upside-down U shape.</p>
  
  <p>Justin's note is clearly saying the same thing as what I wrote above, with added note that the maximum of the curve will happen when x = 2, and Bengio's note is saying the same thing without telling you where the peak could be.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-28T03:42:53.030Z
Answers.LastActivityDate: 2017-08-28T03:42:53.030Z
Answers.UserDisplayName: DiveIntoML
Body: <p>In this <a href="https://people.cs.umass.edu/~domke/courses/sml2011/08autodiff_nnets.pdf" rel="nofollow noreferrer">note</a> Justin Domke says that</p>
  
  <blockquote>
    <p>In practice, neural networks seem to usually find a reasonable solution when the number of layers is not too large, but find poor solutions when using more than, say, 2 hidden layers.</p>
  </blockquote>
  
  <p>But in <a href="https://qr.ae/pNrt0p" rel="nofollow noreferrer">Bengio's remark</a>, he says </p>
  
  <blockquote>
    <p>Very simple. Just keep adding layers until the test error does not improve anymore.</p>
  </blockquote>
  
  <p>There seems to be a conflict. Can anyone explain why they suggest differently? Or am I missing something?</p>
  
Tags: <neural-networks><machine-learning><hidden-layers>
Title: Are these statements about the performance of neural networks as a function of the number of hidden layers contradictory?
UserDisplayName: Ming
ThreadText: # Are these statements about the performance of neural networks as a function of the number of hidden layers contradictory?
  
  (Asked by: username_0 on 2017-08-28)
  
  In this [note](https://people.cs.umass.edu/~domke/courses/sml2011/08autodiff_nnets.pdf) Justin Domke says that
  
  > In practice, neural networks seem to usually find a reasonable solution when the number of layers is not too large, but find poor solutions when using more than, say, 2 hidden layers.
  
  But in [Bengio's remark](https://qr.ae/pNrt0p), he says
  
  > Very simple. Just keep adding layers until the test error does not improve anymore.
  
  There seems to be a conflict. Can anyone explain why they suggest differently? Or am I missing something?
  
  **Tags:** neural-networks, machine-learning, hidden-layers
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2017-08-28. Score: 2
  
  There are **many** problems requiring more than two hidden layers. Randomly select a recent Google journal paper on deep learning, you'll see their network could have something like 5 (or more) hidden layers.
  
  Justin Domke wrote his notes for students, so he probably tried to make his points as simple as possible. For a "typical" machine learning problem that students would most likely work on, two hidden layers should be sufficient. But that doesn't add up for a real practical problem. "Deep" learning usually mean more than two hidden layers.
  
  Number of hidden layers is network design that nobody knows for sure. Yoshua Bengio's suggestion is common and simple. It's not a mathmatic proof, but simply a guideline if you don't know what to do. You just repeat and repeat, until you see the test error no longer improve.
  
  
  ---
  
  ### Answer by username_2 on 2020-04-12. Score: 0
  
  Your first link is from 2011, which essentially predates the current deep learning explosion. In the many years that have since passed (AlexNet 2012, ResNet 2015) we have since found that if you keep adding layers, we generally do see improved performance.
  
  [![enter image description here](https://i.sstatic.net/chMZQ.png)](https://i.sstatic.net/chMZQ.png)
  
  This is due to improved training techniques and optimization breakthroughs (residual connections, ReLU, dropout etc.). But do note that the result can be diminishing. In particular, take a look at [Deep Equillibrium Models](https://arxiv.org/pdf/1909.01377.pdf), which essentially allow us to train (in the limit equivalence) *infinite depth* neural networks.
  
  
  ---
  
  ### Answer by username_3 on 2017-08-28. Score: -1
  
  In fact they are telling the same thing: Plot the x-axis as the number of hidden layers, and the y-axis as the performance (e.g. classification accuracy), then this curve will have an upside-down U shape.
  
  Justin's note is clearly saying the same thing as what I wrote above, with added note that the maximum of the curve will happen when x = 2, and Bengio's note is saying the same thing without telling you where the peak could be.
  
  
  ---
  
  