Answers.Body: <p>Note: The following statements are about feed-forward neural networks. If you're interested in something else, please let me know.</p>
  
  <p>There is a paper which proofs that one hidden layer is enough. So theoretically you could represent any network with many layers with a single hidden layer. I don't remember the title, though. Also I don't remember how it scaled (how many additional nodes you need per added layer).</p>
  
  <p>In practice, it is a very different story anyway: </p>
  
  <ol>
  <li>You need to train the network in an acceptable wall-clock time,</li>
  <li>able to fit stuff in memory</li>
  <li>You have limitations on maximum time for one inference</li>
  <li>You need to find the weights by a training algorithm</li>
  </ol>
  
  <p>Anecdotally, I can say wider networks are usually have more problems with overfitting / memory limitation problems while deeper networks have more problems with maximum inference time.</p>
  
  <p>There is a nice paper from Microsoft where they trained the first super deep network. Many papers followed to that one:</p>
  
  <blockquote>
    <p>He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.</p>
  </blockquote>
  
  <p>One main claim was that deeper networks learn more complex features. I don't remember how much of a problem overfitting was. Another important insight is that <strong>skip connections are critical for really deep networks</strong> (> 20 learning layers).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-12T16:25:24.483Z
Answers.Text: "I don't remember the title, though", maybe, before posting this answer, you could have looked it up, as this answer, if good enough, could serve as reference? Anyway, you can still add this detail to your answer. I can say a similar argument regarding your  "I don't remember how much of a problem overfitting was". I am asking you to cite and give the actual numbers because this is still an on-going area of research and many things are still poorly understood. Maybe this post may be useful to you: https://stats.stackexchange.com/q/182734/82135.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-12T16:42:30.933Z
Answers.Text: nbro, you seem to forget that I'm using my free time to answer this question. If you want me to put more effort in it,  ask directly and politely for it. You leave a lot of comments here in this direction, that don't provide any value.
Answers.UserDisplayName: Martin Thoma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-12T16:47:17.167Z
Answers.Text: If you want to know how much overfitting is a problem for deep networks for a specific dataset in practice, I encourage you to try it. It's not hard, only time consuming. Please also don't forget to share the results if you do so.
Answers.UserDisplayName: Martin Thoma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-12T16:49:22.460Z
Answers.Text: I am also using my time to review questions and answers of this website, so that to make the website improve in quality. If you took the effort to write a 10-minute answer (or whatever), then I don't understand why would it be a problem to also look for a reference (1 or 2 minutes more). It's not about free time or not. It's about providing reliability or do nothing.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-12T16:54:29.420Z
Answers.Text: Then you should use your time more productive. Currently, for the comments I read here, I'd say you're a troll. From your six answers on this site I read two so far. One was good (+1), the other one has issues similar to what you criticize. You should first provide actual value before you make this site less attractive for active contributors. I showed you a way to do it. It's now in you. I'll also not answer to more contents here. If you want to chat, open a chat.
Answers.UserDisplayName: Martin Thoma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-12T16:56:34.117Z
Answers.Text: If I was a regular guy, I could be offended by "you're a troll". No, I am not a troll. Tell which of my answers suffer from the weakness mentioned in my criticisms. You will understand that you got this wrong and that you talked without reading my actual answers. 3 answers out of 6 use citations. The others do not cite anything because that's pretty much regular knowledge. If they didn't receive many upvotes, it's because people either didn't see them, they didn't understand the contents (because they do not have the necessary knowledge to understand them), or they didn't want to give me rep.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-12T15:57:07.817Z
Answers.LastActivityDate: 2018-11-12T16:02:26.067Z
Answers.LastEditDate: 2018-11-12T16:02:26.067Z
Answers.UserDisplayName: Martin Thoma
Body: <p>How is a feed-forward neural network with few hidden layers and lots of nodes in those hidden layers different from a network with a lot of hidden layers but relatively lesser nodes in those hidden layers?</p>
  
Tags: <neural-networks><machine-learning><deep-learning><deep-neural-networks><feedforward-neural-networks>
Title: Significance of depth of a deep neural network
UserDisplayName: Prayush Dawda
ThreadText: # Significance of depth of a deep neural network
  
  (Asked by: username_0 on 2018-11-10)
  
  How is a feed-forward neural network with few hidden layers and lots of nodes in those hidden layers different from a network with a lot of hidden layers but relatively lesser nodes in those hidden layers?
  
  **Tags:** neural-networks, machine-learning, deep-learning, deep-neural-networks, feedforward-neural-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-11-12. Score: 1
  
  Note: The following statements are about feed-forward neural networks. If you're interested in something else, please let me know.
  
  There is a paper which proofs that one hidden layer is enough. So theoretically you could represent any network with many layers with a single hidden layer. I don't remember the title, though. Also I don't remember how it scaled (how many additional nodes you need per added layer).
  
  In practice, it is a very different story anyway:
  
  1. You need to train the network in an acceptable wall-clock time,
  2. able to fit stuff in memory
  3. You have limitations on maximum time for one inference
  4. You need to find the weights by a training algorithm
  
  Anecdotally, I can say wider networks are usually have more problems with overfitting / memory limitation problems while deeper networks have more problems with maximum inference time.
  
  There is a nice paper from Microsoft where they trained the first super deep network. Many papers followed to that one:
  
  > He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
  
  One main claim was that deeper networks learn more complex features. I don't remember how much of a problem overfitting was. Another important insight is that **skip connections are critical for really deep networks** (> 20 learning layers).
  
  > **Comments:**
  > **username_2**: "I don't remember the title, though", maybe, before posting this answer, you could have looked it up, as this answer, if good enough, could serve as reference? Anyway, you can still add this detail to your answer. I can say a similar argument regarding your  "I don't remember how much of a problem overfitting was". I am asking you to cite and give the actual numbers because this is still an on-going area of research and many things are still poorly understood. Maybe this post may be useful to you: https://stats.stackexchange.com/q/182734/82135.  
  > **username_1**: nbro, you seem to forget that I'm using my free time to answer this question. If you want me to put more effort in it,  ask directly and politely for it. You leave a lot of comments here in this direction, that don't provide any value.  
  > **username_1**: If you want to know how much overfitting is a problem for deep networks for a specific dataset in practice, I encourage you to try it. It's not hard, only time consuming. Please also don't forget to share the results if you do so.  
  > **username_2**: I am also using my time to review questions and answers of this website, so that to make the website improve in quality. If you took the effort to write a 10-minute answer (or whatever), then I don't understand why would it be a problem to also look for a reference (1 or 2 minutes more). It's not about free time or not. It's about providing reliability or do nothing.  
  > **username_1**: Then you should use your time more productive. Currently, for the comments I read here, I'd say you're a troll. From your six answers on this site I read two so far. One was good (+1), the other one has issues similar to what you criticize. You should first provide actual value before you make this site less attractive for active contributors. I showed you a way to do it. It's now in you. I'll also not answer to more contents here. If you want to chat, open a chat.  
  > **username_2**: If I was a regular guy, I could be offended by "you're a troll". No, I am not a troll. Tell which of my answers suffer from the weakness mentioned in my criticisms. You will understand that you got this wrong and that you talked without reading my actual answers. 3 answers out of 6 use citations. The others do not cite anything because that's pretty much regular knowledge. If they didn't receive many upvotes, it's because people either didn't see them, they didn't understand the contents (because they do not have the necessary knowledge to understand them), or they didn't want to give me rep.  
  
  ---
  
  