Answers.Body: <p>This is akin to asking &quot;Why do we need more than one instance of sine to represent any repeating function&quot; or &quot;why can't we represent any polynomial with an equivalent polynomial of just the first degree?&quot;  There are many, many problems... I'd even want to say most... that will require more than one layer to solve because the higher dimensional relationships cannot be well represented by just one layer.  This is not to say that the theorem is wrong, but consider the applied aspects.  We can approximate any continuous function, but that might require a single layer that is infinitely wide, however that same function might be approximated by a deep network having only a few dozen neurons.</p>
  <p>However, this is not to say that many networks could not be represented with networks that perform at least as well, or perhaps even better, by simpler networks of fewer layers/neurons.  There is active research into how to generalize this.</p>
  <p>Ultimately, non-trivial problems often require an empirical approach in this space currently because there is no general solution to &quot;learning.&quot;</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-21T10:36:16.917Z
Answers.Text: I think what you have answered is not correct in fact due to universal approximation theorem, we can approximate any higher dimensional function with only one hidden layer.
Answers.UserDisplayName: Prakhar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-21T11:21:01.343Z
Answers.Text: @prakharnigam I think that the idea that this answer tries to convey is correct. You may be able to approximate your function with only one sine-like function by increasing e.g. the frequency, but you may do that more efficiently with a combination of multiple functions (and that's why using always a 1-hidden layer net is not typically done). Note that the universal approximation theorem (by Cybenko) tells you that you can **approximate** any continuous function, but provided that the number of neurons in the hidden layer is **sufficiently large**.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-21T11:34:09.027Z
Answers.Text: @nbro I was asking, whether increasing layers is more efficient or adding neurons and why. That answer was a very generic one, of course i know what he is trying to convey but it is not rigorous and incorrect in some parts like saying " that will require more than one layer to solve because the higher dimensional relationships cannot be well represented by just one layer", well every higher dimensional relationship that is continuous can be represented by one hidden layer.
Answers.UserDisplayName: Prakhar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-21T11:56:10.707Z
Answers.Text: As I stated above, the UAT tells you that, _in theory_, you can approximate a continuous function with a 1-hidden layer (with an arbitrary width) neural network, but it doesn't tell you how to do it _in practice_. I don't know the exact mathematical answer to your question, but we can say that, _in practice_, in many cases, neural networks with more than 1 hidden layer are used (probably because they are more effective). You may be interested in [this post](https://ai.stackexchange.com/q/20680/2444) (and the related/linked posts and papers).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-21T13:55:09.017Z
Answers.Text: @nbro Thanks! That post along with papers linked might be helpful.
Answers.UserDisplayName: Prakhar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-21T14:29:52.333Z
Answers.Text: @prakharnigam I just read a paper last week that I'm having trouble finding that might be helpful as well...  The topic was a new approach toward leaf and edge pruning in networks for graph connection pruning of weights and neuron pruning.. This is connected to the tendency for only a handful of neurons and the connections to following layers driving the majority of the accuracy in a network..  That isn't your question, but I suspect you'll find it is very much related.
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-21T15:26:20.990Z
Answers.Text: thanks for recommending, will see it too.
Answers.UserDisplayName: Prakhar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-20T22:25:26.040Z
Answers.LastActivityDate: 2021-05-22T10:35:11.137Z
Answers.LastEditDate: 2021-05-22T10:35:11.137Z
Answers.UserDisplayName: David Hoelzer
Body: <p>I have read about the universal approximation theorem. So, why do we need more than 1 layer? Is it somehow computationally efficient to add layers instead of more neurons in the hidden layer?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-20T22:36:33.803Z
Comments.Text: I would highly recommend watching this introductory lecture by [DeepMind x UCL](https://www.youtube.com/watch?v=FBggC-XVF4M) (34:00 - 40:00) which explains the Universal Approximation Theorem in an intuitive way.
Comments.UserDisplayName: orion_ix
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-21T00:31:55.377Z
Comments.Text: @orion_ix wow, that video is fantastic! It explained why we go deep instead of wide in a very insightful way thanks for recommendation!
Comments.UserDisplayName: Prakhar
Tags: <neural-networks><machine-learning><deep-learning><deep-neural-networks><universal-approximation-theorems>
Title: Do we ever need more then 1 hidden layer in a binary classification problem with ANNs? If yes why?
UserDisplayName: Prakhar
ThreadText: # Do we ever need more then 1 hidden layer in a binary classification problem with ANNs? If yes why?
  
  (Asked by: username_0 on 2021-05-20)
  
  I have read about the universal approximation theorem. So, why do we need more than 1 layer? Is it somehow computationally efficient to add layers instead of more neurons in the hidden layer?
  
  **Tags:** neural-networks, machine-learning, deep-learning, deep-neural-networks, universal-approximation-theorems
  > **Comments:**
  > **username_1**: I would highly recommend watching this introductory lecture by [DeepMind x UCL](https://www.youtube.com/watch?v=FBggC-XVF4M) (34:00 - 40:00) which explains the Universal Approximation Theorem in an intuitive way.  
  > **username_0**: @orion_ix wow, that video is fantastic! It explained why we go deep instead of wide in a very insightful way thanks for recommendation!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-05-20. Score: 2
  
  This is akin to asking "Why do we need more than one instance of sine to represent any repeating function" or "why can't we represent any polynomial with an equivalent polynomial of just the first degree?" There are many, many problems... I'd even want to say most... that will require more than one layer to solve because the higher dimensional relationships cannot be well represented by just one layer. This is not to say that the theorem is wrong, but consider the applied aspects. We can approximate any continuous function, but that might require a single layer that is infinitely wide, however that same function might be approximated by a deep network having only a few dozen neurons.
  
  However, this is not to say that many networks could not be represented with networks that perform at least as well, or perhaps even better, by simpler networks of fewer layers/neurons. There is active research into how to generalize this.
  
  Ultimately, non-trivial problems often require an empirical approach in this space currently because there is no general solution to "learning."
  
  > **Comments:**
  > **username_0**: I think what you have answered is not correct in fact due to universal approximation theorem, we can approximate any higher dimensional function with only one hidden layer.  
  > **username_3**: @prakharnigam I think that the idea that this answer tries to convey is correct. You may be able to approximate your function with only one sine-like function by increasing e.g. the frequency, but you may do that more efficiently with a combination of multiple functions (and that's why using always a 1-hidden layer net is not typically done). Note that the universal approximation theorem (by Cybenko) tells you that you can **approximate** any continuous function, but provided that the number of neurons in the hidden layer is **sufficiently large**.  
  > **username_0**: @nbro I was asking, whether increasing layers is more efficient or adding neurons and why. That answer was a very generic one, of course i know what he is trying to convey but it is not rigorous and incorrect in some parts like saying " that will require more than one layer to solve because the higher dimensional relationships cannot be well represented by just one layer", well every higher dimensional relationship that is continuous can be represented by one hidden layer.  
  > **username_3**: As I stated above, the UAT tells you that, _in theory_, you can approximate a continuous function with a 1-hidden layer (with an arbitrary width) neural network, but it doesn't tell you how to do it _in practice_. I don't know the exact mathematical answer to your question, but we can say that, _in practice_, in many cases, neural networks with more than 1 hidden layer are used (probably because they are more effective). You may be interested in [this post](https://ai.stackexchange.com/q/20680/2444) (and the related/linked posts and papers).  
  > **username_0**: @nbro Thanks! That post along with papers linked might be helpful.  
  > **username_2**: @prakharnigam I just read a paper last week that I'm having trouble finding that might be helpful as well...  The topic was a new approach toward leaf and edge pruning in networks for graph connection pruning of weights and neuron pruning.. This is connected to the tendency for only a handful of neurons and the connections to following layers driving the majority of the accuracy in a network..  That isn't your question, but I suspect you'll find it is very much related.  
  > **username_0**: thanks for recommending, will see it too.  
  
  ---
  
  