Answers.Body: <p>A feed forward neural network without hidden nodes can only find linear decision boundaries. However, most of the time you need non-linear decision boundaries. Hence you need hidden nodes with a non-linear activation function. The more hidden nodes you have, the more data you need to find good parameters, but the more complex decision boundaries you can find.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-11-01T03:44:59.577Z
Answers.Text: It appears that the topic is much more complex than I initially thought. What field of study is this in, and is there a name for this specific aspect of neural networking?
Answers.UserDisplayName: user289661
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-11-01T10:41:42.003Z
Answers.Text: @user289661 This is one of the most basic aspects of artificial neural networks. It's in machine learning.
Answers.UserDisplayName: Martin Thoma
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-11-01T21:12:11.620Z
Answers.Text: And you generally want more than one hidden layer, because while a sufficiently large network can capture any function to an arbitrary precision, for certain computations a shallow network would need exponentially more nodes than a one layer deeper network that does the same.
Answers.UserDisplayName: Peteris
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-11-01T22:05:28.260Z
Answers.Text: @Peteris While I agree that you generally want more than one hidden layer, I wonder if you have a source for the claim that a shallow network needs exponentially more nodes than a one layer deeper network.
Answers.UserDisplayName: Martin Thoma
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-11-01T23:17:17.260Z
Answers.Text: @MartinThoma https://arxiv.org/pdf/1603.00988.pdf or https://arxiv.org/pdf/1509.08101.pdf are some sources; the particular proofs of course are different for different types of functions; It generally starts with Hastad 1987 "Computational limitations of small-depth circuits" and in general this concept is the counterpoint to the universal approximator theorem - while yes, even the simplest architectures can implement anything, for certain functions they require an unreasonable amount of resources.
Answers.UserDisplayName: Peteris
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-11-01T23:23:39.170Z
Answers.Text: @MartinThoma http://deeplearning.cs.cmu.edu/pdfs/Delalleau_Bengio_2011.pdf can also be relevant.
Answers.UserDisplayName: Peteris
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-10-23T22:35:38.523Z
Answers.LastActivityDate: 2016-10-25T08:00:48.090Z
Answers.LastEditDate: 2016-10-25T08:00:48.090Z
Answers.UserDisplayName: Martin Thoma
Answers.Body: <p>Normally one node/layer applies linear fitting of the the input to the hypothesis, in other words uses linear function (<span class="math-container">$y = ax + b$</span>). Adding layers chains liner functions, potentially allowing fitting higher order functions. A great explanation can be found <a href="http://colah.github.io/posts/2015-01-Visualizing-Representations/" rel="nofollow noreferrer">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2016-10-23T18:49:12.407Z
Answers.LastActivityDate: 2018-12-28T19:34:51.137Z
Answers.LastEditDate: 2018-12-28T19:34:51.137Z
Answers.UserDisplayName: Iliyan Bobev
Body: <p>If I have a set of sensory nodes taking in information and a set of "action nodes" which determine the behavior of my robot, why do I need hidden nodes between them when I can let all sensory nodes affect all action nodes?</p>
  
  <p>(This is in the context of evolving neural network)</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-10-24T06:55:20.593Z
Comments.Text: Look for the idea called "Hierarchical representation"
Comments.UserDisplayName: Ankur
Tags: <neural-networks><convolutional-neural-networks><evolutionary-algorithms>
Title: What is the purpose of hidden nodes in neural network?
UserDisplayName: user289661
ThreadText: # What is the purpose of hidden nodes in neural network?
  
  (Asked by: username_0 on 2016-10-23)
  
  If I have a set of sensory nodes taking in information and a set of "action nodes" which determine the behavior of my robot, why do I need hidden nodes between them when I can let all sensory nodes affect all action nodes?
  
  (This is in the context of evolving neural network)
  
  **Tags:** neural-networks, convolutional-neural-networks, evolutionary-algorithms
  > **Comments:**
  > **username_1**: Look for the idea called "Hierarchical representation"  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2016-10-23. Score: 5
  
  A feed forward neural network without hidden nodes can only find linear decision boundaries. However, most of the time you need non-linear decision boundaries. Hence you need hidden nodes with a non-linear activation function. The more hidden nodes you have, the more data you need to find good parameters, but the more complex decision boundaries you can find.
  
  > **Comments:**
  > **username_0**: It appears that the topic is much more complex than I initially thought. What field of study is this in, and is there a name for this specific aspect of neural networking?  
  > **username_2**: @user289661 This is one of the most basic aspects of artificial neural networks. It's in machine learning.  
  > **username_3**: And you generally want more than one hidden layer, because while a sufficiently large network can capture any function to an arbitrary precision, for certain computations a shallow network would need exponentially more nodes than a one layer deeper network that does the same.  
  > **username_2**: @Peteris While I agree that you generally want more than one hidden layer, I wonder if you have a source for the claim that a shallow network needs exponentially more nodes than a one layer deeper network.  
  > **username_3**: @MartinThoma https://arxiv.org/pdf/1603.00988.pdf or https://arxiv.org/pdf/1509.08101.pdf are some sources; the particular proofs of course are different for different types of functions; It generally starts with Hastad 1987 "Computational limitations of small-depth circuits" and in general this concept is the counterpoint to the universal approximator theorem - while yes, even the simplest architectures can implement anything, for certain functions they require an unreasonable amount of resources.  
  > **username_3**: @MartinThoma http://deeplearning.cs.cmu.edu/pdfs/Delalleau_Bengio_2011.pdf can also be relevant.  
  
  ---
  
  ### Answer by username_4 on 2016-10-23. Score: 1
  
  Normally one node/layer applies linear fitting of the the input to the hypothesis, in other words uses linear function ($y = ax + b$). Adding layers chains liner functions, potentially allowing fitting higher order functions. A great explanation can be found [here](http://colah.github.io/posts/2015-01-Visualizing-Representations/).
  
  
  ---
  
  