Answers.Body: <h2>Deeper models can have advantages (in certain cases)</h2>
  
  <p>Most people will answer "yes" to your question, see e.g. <a href="https://stats.stackexchange.com/q/222883/82135">Why are neural networks becoming deeper, but not wider?</a> and <a href="https://math.stackexchange.com/q/3147754/168764">Why do deep neural networks work well?</a>. </p>
  
  <p>In fact, there are cases where deep neural networks have certain advantages compared to shallow ones. For example, see the following papers </p>
  
  <ul>
  <li><a href="http://proceedings.mlr.press/v49/eldan16.pdf" rel="noreferrer">The Power of Depth for Feedforward Neural Networks</a> (2016) by Ronen Eldan and Ohad Shamir</li>
  <li><a href="https://arxiv.org/pdf/1602.04485.pdf" rel="noreferrer">Benefits of depth in neural networks</a> (2016) by Matus Telgarsky.</li>
  <li><a href="https://arxiv.org/pdf/1610.09887.pdf" rel="noreferrer">Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks</a> (2017) by Safran and Shamir</li>
  <li><a href="https://arxiv.org/pdf/1709.05289.pdf" rel="noreferrer">Optimal approximation of piecewise smooth functions using deep ReLU neural networks</a> (2018) by Petersen and Voigtlaender</li>
  </ul>
  
  <h2>What about the width?</h2>
  
  <p>The following papers may be relevant</p>
  
  <ul>
  <li><a href="https://arxiv.org/abs/1605.07146" rel="noreferrer">Wide Residual Networks</a> (2017) by Sergey Zagoruyko and Nikos Komodakis</li>
  <li><a href="http://papers.nips.cc/paper/7203-the-expressive-power-of-neural-networks-a-view-from-the-width" rel="noreferrer">The Expressive Power of Neural Networks: A View from the Width</a> (2017) by Zhou Lu et al.</li>
  </ul>
  
  <h2>Bigger models have bigger capacity but also have disadvantages</h2>
  
  <p>Vladimir Vapnik (co-inventor of VC theory and SVMs, and one of the most influential contributors to learning theory), who is <a href="https://www.youtube.com/watch?v=Ow25mjFjSmg" rel="noreferrer">not a fan of neural networks</a>, will probably tell you that you should look for the smallest model (set of functions) that is consistent with your data (i.e. an admissible set of functions).  </p>
  
  <p>For example, watch this podcast <a href="https://www.youtube.com/watch?v=STFcvzoxVw4" rel="noreferrer">Vladimir Vapnik: Statistical Learning | Artificial Intelligence (AI) Podcast</a> (2018), where he says this. His new learning theory framework based on statistical invariants and predicates can be found in the paper <a href="https://link.springer.com/article/10.1007/s10994-018-5742-0" rel="noreferrer">Rethinking statistical learning theory: learning using statistical invariants</a> (2019). You should also read <a href="https://www.learningtheory.org/learning-has-just-started-an-interview-with-prof-vladimir-vapnik/" rel="noreferrer">"Learning Has Just Started" – an interview with Prof. Vladimir Vapnik</a> (2014).</p>
  
  <p>Bigger models have a bigger capacity (i.e. a bigger VC dimension), which means that you will more likely <strong>overfit the training data</strong>, i.e., the model may not really be able to generalize to unseen data. So, in order not to overfit, models with more parameters (and thus capacity) will also require more data. You should also ask yourself why people use <strong>regularisation techniques</strong>. </p>
  
  <p>In practice, models that achieve state-of-the-art performance can be very deep, but they are also <strong>computationally inefficient to train</strong> and they <strong>require huge amounts of training data</strong> (either manually labeled or automatically generated). </p>
  
  <p>Moreover, there are many other technical complications with deeper neural networks, for example, problems such as the <strong>vanishing (and exploding) gradient problem</strong>. </p>
  
  <h2>Complex tasks may not require bigger models</h2>
  
  <p>Some people will tell you that you require deep models because, empirically, some deep models have achieved state-of-the-art results, but that's probably because we haven't found cleverer and more efficient ways of solving these problems. </p>
  
  <p>Therefore, I would not say that "complex tasks" (whatever the definition is) necessarily require deeper or, in general, bigger models. While designing our models, it may be a good idea to always keep in mind principles like Occam's razor!</p>
  
  <h2>A side note</h2>
  
  <p>As a side note, I think that more people should focus more on the mathematical aspects of machine learning, i.e. computational and statistical learning theory. There are too many practitioners, who don't really understand the underlying learning theory, and too few theorists, and the progress could soon stagnate because of a lack of understanding of the underlying mathematical concepts.</p>
  
  <p>To give you a more concrete idea of the current mentality of the deep learning community, in <a href="https://www.youtube.com/watch?v=9EN_HoEk3KY" rel="noreferrer">this lesson</a>, a person like Ilya Sutskever, who is considered an "important and leading" researcher in deep learning, talks about NP-complete problems as if he doesn't really know what he's talking about. NP-complete problems aren't just "hard problems". NP-completeness has a very specific definition in computational complexity theory!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-28T05:39:24.490Z
Answers.Text: I'm having trouble following this answer. Are you saying that the answer might be "yes" in practice, but in theory we might find ways to solve more complex problems with shallower nets?
Answers.UserDisplayName: The Guy with The Hat
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-28T12:24:56.847Z
Answers.Text: @TheGuywithTheHat Roughly, yes. Current methods that have solved "complex tasks" aren't particularly clever and don't scale well. If you have to use huge amounts of data or spend so much time and resources to train a net, or if training pollutes the environment so much, you're probably not doing the cleverest thing. We still don't understand many things about NNs. With a more solid understanding of learning theory, we will be able to come up with cleverer solutions.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-28T20:43:58.427Z
Answers.Text: Actually there is evidence that increasing the number of parameters can, paradoxically, reduce the overfitting error. One paper I looked at presented a theory that the smallest model that can be trained to fit the data represents a local maximum in generalization error: you either want a smaller model (to reduce VC dimension) or a larger model (for reasons that are not entirely clear; perhaps an overparametrized model ends up behaving a bit like an ensemble when trained with current methods). But mostly this chain of reasoning is concerned with the width rather than the depth of models.
Answers.UserDisplayName: Charles Staats
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-28T20:45:54.547Z
Answers.Text: @CharlesStaats Can you provide the title of and link to the paper?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-28T23:43:26.137Z
Answers.Text: [DEEP DOUBLE DESCENT:
  WHERE BIGGER MODELS AND MORE DATA HURT](https://arxiv.org/pdf/1912.02292.pdf)
Answers.UserDisplayName: Charles Staats
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T15:33:27.723Z
Answers.LastActivityDate: 2020-05-12T12:49:40.957Z
Answers.LastEditDate: 2020-05-12T12:49:40.957Z
Answers.UserDisplayName: nbro
Answers.Body: <p>My experience from a tactical standpoint is to start out with a smaller simple model first. Train the model and observe the training accuracy and validation loss and validation accuracy. My observation is that, to be a good model, your training accuracy should achieve a value of at least 95%. If it does not, then try to optimize some of the hyper-parameters. If the training accuracy does not improve, then you may try to incrementally add more complexity to the model. As you add more complexity the risk of overfitting, vanishing or exploding gradients becomes higher. </p>
  
  <p>You can detect overfitting by monitoring the validation loss. If as the model accuracy goes up the validation loss on later epochs starts to go up you are overfitting. At that point, you will have to take remedial action in your model like adding dropout layers and use regularizers. Keras documentation is <a href="https://keras.io/regularizers/" rel="nofollow noreferrer">here</a>.</p>
  
  <p>As pointed out in <a href="https://ai.stackexchange.com/a/20689/2444">the answer by nbro</a>, the theory addressing this issue is complex. I highly recommend the excellent tutorial on this subject which can be found on YouTube <a href="https://www.youtube.com/watch?v=mbyG85GZ0PI&amp;list=PLnIDYuXHkit4LcWjDe0EwlE57WiGlBs08" rel="nofollow noreferrer">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T18:28:33.640Z
Answers.LastActivityDate: 2020-04-27T20:32:12.897Z
Answers.LastEditDate: 2020-04-27T20:32:12.897Z
Answers.UserDisplayName: Gerry P
Answers.Body: <p>Deeper networks have more learning capacity in the sense that they can fit to more complex data. But at the same time, they are also more prone to overfitting the training data and therefore fails to generalize to the test set.   </p>
  
  <p>Apart from overfitting, exploding/vanishing gradients is another problem which hampers convergence. This can be addressed by normalizing the initialization and normalizing the intermediate layers. Then you can do backpropagation with stochastic gradient descent (SGD).   </p>
  
  <p>When deeper networks are able to converge, another problem of 'degradation' has been detected. The accuracy saturates and then starts to degrade. This is not caused by overfitting. In fact, adding more layers here leads to higher training error. A possible fix is to use ResNets (residual networks), which have been shown to decrease 'degradation'</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T15:49:09.493Z
Answers.LastActivityDate: 2020-04-27T15:49:09.493Z
Answers.UserDisplayName: Abhijith S Raj
Answers.Body: <p>Speaking <em>very</em> generally, I would say that with the current state of machine learning, a "more complicated" task requires <em>more trainable parameters</em>. You can increase parameter count by either increasing width and also by increasing depth. Again, speaking <em>very generally</em>, I would say that in practice, people have found more success by increasing depth than by increasing width.</p>
  
  <p>However, this depends a lot on what you mean by "more complicated". I would argue that generating something is a fundamentally more complicated problem than just identifying something. However, a GAN to generate a 4-pixel image will probably be far more shallow than the shallowest ImageNet network.</p>
  
  <p>One could also make an argument that the <em>definition</em> of complexity of a deep learning task is "more layers needed == more complicated", in which case it's obvious that by definition, a more complicated task requires a deeper net.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-28T04:58:36.503Z
Answers.LastActivityDate: 2020-04-28T05:05:21.250Z
Answers.LastEditDate: 2020-04-28T05:05:21.250Z
Answers.UserDisplayName: The Guy with The Hat
Body: <p>I know it's not an exact science. But would you say that generally for more complicated tasks, deeper nets are required?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-28T23:13:52.660Z
Comments.Text: The gist is that you can't just pick any numbers for width and depth and expect optimal performance. Picking the right size network is somewhat trial-and-error, and somewhat of an art.
Comments.UserDisplayName: MaxW
Tags: <neural-networks><deep-learning><architecture>
Title: Should neural nets be deeper the more complex the learning problem is?
UserDisplayName: Gilad Deutsch
ThreadText: # Should neural nets be deeper the more complex the learning problem is?
  
  (Asked by: username_0 on 2020-04-27)
  
  I know it's not an exact science. But would you say that generally for more complicated tasks, deeper nets are required?
  
  **Tags:** neural-networks, deep-learning, architecture
  
  > **Comments:**
  > **username_1**: The gist is that you can't just pick any numbers for width and depth and expect optimal performance. Picking the right size network is somewhat trial-and-error, and somewhat of an art.  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_2 on 2020-04-27. Score: 15
  
  ## Deeper models can have advantages (in certain cases)
  
  Most people will answer "yes" to your question, see e.g. [Why are neural networks becoming deeper, but not wider?](https://stats.stackexchange.com/q/222883/82135) and [Why do deep neural networks work well?](https://math.stackexchange.com/q/3147754/168764).
  
  In fact, there are cases where deep neural networks have certain advantages compared to shallow ones. For example, see the following papers
  
  * [The Power of Depth for Feedforward Neural Networks](http://proceedings.mlr.press/v49/eldan16.pdf) (2016) by Ronen Eldan and Ohad Shamir
  * [Benefits of depth in neural networks](https://arxiv.org/pdf/1602.04485.pdf) (2016) by Matus Telgarsky.
  * [Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks](https://arxiv.org/pdf/1610.09887.pdf) (2017) by Safran and Shamir
  * [Optimal approximation of piecewise smooth functions using deep ReLU neural networks](https://arxiv.org/pdf/1709.05289.pdf) (2018) by Petersen and Voigtlaender
  
  ## What about the width?
  
  The following papers may be relevant
  
  * [Wide Residual Networks](https://arxiv.org/abs/1605.07146) (2017) by Sergey Zagoruyko and Nikos Komodakis
  * [The Expressive Power of Neural Networks: A View from the Width](http://papers.nips.cc/paper/7203-the-expressive-power-of-neural-networks-a-view-from-the-width) (2017) by Zhou Lu et al.
  
  ## Bigger models have bigger capacity but also have disadvantages
  
  Vladimir Vapnik (co-inventor of VC theory and SVMs, and one of the most influential contributors to learning theory), who is [not a fan of neural networks](https://www.youtube.com/watch?v=Ow25mjFjSmg), will probably tell you that you should look for the smallest model (set of functions) that is consistent with your data (i.e. an admissible set of functions).
  
  For example, watch this podcast [Vladimir Vapnik: Statistical Learning | Artificial Intelligence (AI) Podcast](https://www.youtube.com/watch?v=STFcvzoxVw4) (2018), where he says this. His new learning theory framework based on statistical invariants and predicates can be found in the paper [Rethinking statistical learning theory: learning using statistical invariants](https://link.springer.com/article/10.1007/s10994-018-5742-0) (2019). You should also read ["Learning Has Just Started" – an interview with Prof. Vladimir Vapnik](https://www.learningtheory.org/learning-has-just-started-an-interview-with-prof-vladimir-vapnik/) (2014).
  
  Bigger models have a bigger capacity (i.e. a bigger VC dimension), which means that you will more likely **overfit the training data**, i.e., the model may not really be able to generalize to unseen data. So, in order not to overfit, models with more parameters (and thus capacity) will also require more data. You should also ask yourself why people use **regularisation techniques**.
  
  In practice, models that achieve state-of-the-art performance can be very deep, but they are also **computationally inefficient to train** and they **require huge amounts of training data** (either manually labeled or automatically generated).
  
  Moreover, there are many other technical complications with deeper neural networks, for example, problems such as the **vanishing (and exploding) gradient problem**.
  
  ## Complex tasks may not require bigger models
  
  Some people will tell you that you require deep models because, empirically, some deep models have achieved state-of-the-art results, but that's probably because we haven't found cleverer and more efficient ways of solving these problems.
  
  Therefore, I would not say that "complex tasks" (whatever the definition is) necessarily require deeper or, in general, bigger models. While designing our models, it may be a good idea to always keep in mind principles like Occam's razor!
  
  ## A side note
  
  As a side note, I think that more people should focus more on the mathematical aspects of machine learning, i.e. computational and statistical learning theory. There are too many practitioners, who don't really understand the underlying learning theory, and too few theorists, and the progress could soon stagnate because of a lack of understanding of the underlying mathematical concepts.
  
  To give you a more concrete idea of the current mentality of the deep learning community, in [this lesson](https://www.youtube.com/watch?v=9EN_HoEk3KY), a person like Ilya Sutskever, who is considered an "important and leading" researcher in deep learning, talks about NP-complete problems as if he doesn't really know what he's talking about. NP-complete problems aren't just "hard problems". NP-completeness has a very specific definition in computational complexity theory!
  
  > **Comments:**
  > **username_3**: I'm having trouble following this answer. Are you saying that the answer might be "yes" in practice, but in theory we might find ways to solve more complex problems with shallower nets?  
  > **username_2**: @username_3 Roughly, yes. Current methods that have solved "complex tasks" aren't particularly clever and don't scale well. If you have to use huge amounts of data or spend so much time and resources to train a net, or if training pollutes the environment so much, you're probably not doing the cleverest thing. We still don't understand many things about NNs. With a more solid understanding of learning theory, we will be able to come up with cleverer solutions.  
  > **username_4**: Actually there is evidence that increasing the number of parameters can, paradoxically, reduce the overfitting error. One paper I looked at presented a theory that the smallest model that can be trained to fit the data represents a local maximum in generalization error: you either want a smaller model (to reduce VC dimension) or a larger model (for reasons that are not entirely clear; perhaps an overparametrized model ends up behaving a bit like an ensemble when trained with current methods). But mostly this chain of reasoning is concerned with the width rather than the depth of models.  
  > **username_2**: @username_4 Can you provide the title of and link to the paper?  
  > **username_4**: [DEEP DOUBLE DESCENT: WHERE BIGGER MODELS AND MORE DATA HURT](https://arxiv.org/pdf/1912.02292.pdf)  
  
  ---
  
  ### Answer by username_5 on 2020-04-27. Score: 3
  
  My experience from a tactical standpoint is to start out with a smaller simple model first. Train the model and observe the training accuracy and validation loss and validation accuracy. My observation is that, to be a good model, your training accuracy should achieve a value of at least 95%. If it does not, then try to optimize some of the hyper-parameters. If the training accuracy does not improve, then you may try to incrementally add more complexity to the model. As you add more complexity the risk of overfitting, vanishing or exploding gradients becomes higher.
  
  You can detect overfitting by monitoring the validation loss. If as the model accuracy goes up the validation loss on later epochs starts to go up you are overfitting. At that point, you will have to take remedial action in your model like adding dropout layers and use regularizers. Keras documentation is [here](https://keras.io/regularizers/).
  
  As pointed out in [the answer by username_2](https://ai.stackexchange.com/a/20689/2444), the theory addressing this issue is complex. I highly recommend the excellent tutorial on this subject which can be found on YouTube [here](https://www.youtube.com/watch?v=mbyG85GZ0PI&list=PLnIDYuXHkit4LcWjDe0EwlE57WiGlBs08).
  
  
  ---
  
  ### Answer by username_6 on 2020-04-27. Score: 2
  
  Deeper networks have more learning capacity in the sense that they can fit to more complex data. But at the same time, they are also more prone to overfitting the training data and therefore fails to generalize to the test set.
  
  Apart from overfitting, exploding/vanishing gradients is another problem which hampers convergence. This can be addressed by normalizing the initialization and normalizing the intermediate layers. Then you can do backpropagation with stochastic gradient descent (SGD).
  
  When deeper networks are able to converge, another problem of 'degradation' has been detected. The accuracy saturates and then starts to degrade. This is not caused by overfitting. In fact, adding more layers here leads to higher training error. A possible fix is to use ResNets (residual networks), which have been shown to decrease 'degradation'
  
  
  ---
  
  ### Answer by username_3 on 2020-04-28. Score: 1
  
  Speaking *very* generally, I would say that with the current state of machine learning, a "more complicated" task requires *more trainable parameters*. You can increase parameter count by either increasing width and also by increasing depth. Again, speaking *very generally*, I would say that in practice, people have found more success by increasing depth than by increasing width.
  
  However, this depends a lot on what you mean by "more complicated". I would argue that generating something is a fundamentally more complicated problem than just identifying something. However, a GAN to generate a 4-pixel image will probably be far more shallow than the shallowest ImageNet network.
  
  One could also make an argument that the *definition* of complexity of a deep learning task is "more layers needed == more complicated", in which case it's obvious that by definition, a more complicated task requires a deeper net.
  
  
  ---
  
  