<p>A feed forward neural network without hidden nodes can only find linear decision boundaries. However, most of the time you need non-linear decision boundaries. Hence you need hidden nodes with a non-linear activation function. The more hidden nodes you have, the more data you need to find good parameters, but the more complex decision boundaries you can find.</p>

CC BY-SA 3.0
2016-11-01T03:44:59.577Z
It appears that the topic is much more complex than I initially thought. What field of study is this in, and is there a name for this specific aspect of neural networking?
user289661
CC BY-SA 3.0
2016-11-01T10:41:42.003Z
@user289661 This is one of the most basic aspects of artificial neural networks. It's in machine learning.
Martin Thoma
CC BY-SA 3.0
2016-11-01T21:12:11.620Z
And you generally want more than one hidden layer, because while a sufficiently large network can capture any function to an arbitrary precision, for certain computations a shallow network would need exponentially more nodes than a one layer deeper network that does the same.
Peteris
CC BY-SA 3.0
2016-11-01T22:05:28.260Z
@Peteris While I agree that you generally want more than one hidden layer, I wonder if you have a source for the claim that a shallow network needs exponentially more nodes than a one layer deeper network.
Martin Thoma
CC BY-SA 3.0
2016-11-01T23:17:17.260Z
@MartinThoma https://arxiv.org/pdf/1603.00988.pdf or https://arxiv.org/pdf/1509.08101.pdf are some sources; the particular proofs of course are different for different types of functions; It generally starts with Hastad 1987 "Computational limitations of small-depth circuits" and in general this concept is the counterpoint to the universal approximator theorem - while yes, even the simplest architectures can implement anything, for certain functions they require an unreasonable amount of resources.
Peteris
CC BY-SA 3.0
2016-11-01T23:23:39.170Z
@MartinThoma http://deeplearning.cs.cmu.edu/pdfs/Delalleau_Bengio_2011.pdf can also be relevant.
Peteris
CC BY-SA 3.0
2016-10-23T22:35:38.523Z
2016-10-25T08:00:48.090Z
2016-10-25T08:00:48.090Z
Martin Thoma
<p>Normally one node/layer applies linear fitting of the the input to the hypothesis, in other words uses linear function (<span class="math-container">$y = ax + b$</span>). Adding layers chains liner functions, potentially allowing fitting higher order functions. A great explanation can be found <a href="http://colah.github.io/posts/2015-01-Visualizing-Representations/" rel="nofollow noreferrer">here</a>.</p>

CC BY-SA 4.0
2016-10-23T18:49:12.407Z
2018-12-28T19:34:51.137Z
2018-12-28T19:34:51.137Z
Iliyan Bobev
<p>If I have a set of sensory nodes taking in information and a set of "action nodes" which determine the behavior of my robot, why do I need hidden nodes between them when I can let all sensory nodes affect all action nodes?</p>

<p>(This is in the context of evolving neural network)</p>

CC BY-SA 3.0
2016-10-24T06:55:20.593Z
Look for the idea called "Hierarchical representation"
Ankur
<neural-networks><convolutional-neural-networks><evolutionary-algorithms>
What is the purpose of hidden nodes in neural network?
user289661
# What is the purpose of hidden nodes in neural network?

(Asked by: username_0 on 2016-10-23)

If I have a set of sensory nodes taking in information and a set of "action nodes" which determine the behavior of my robot, why do I need hidden nodes between them when I can let all sensory nodes affect all action nodes?

(This is in the context of evolving neural network)

**Tags:** neural-networks, convolutional-neural-networks, evolutionary-algorithms

> **Comments:**
> **username_1**: Look for the idea called "Hierarchical representation"  

---

## 2 answers

### Answer by username_2 on 2016-10-23. Score: 5

A feed forward neural network without hidden nodes can only find linear decision boundaries. However, most of the time you need non-linear decision boundaries. Hence you need hidden nodes with a non-linear activation function. The more hidden nodes you have, the more data you need to find good parameters, but the more complex decision boundaries you can find.

> **Comments:**
> **username_0**: It appears that the topic is much more complex than I initially thought. What field of study is this in, and is there a name for this specific aspect of neural networking?  
> **username_2**: @username_0 This is one of the most basic aspects of artificial neural networks. It's in machine learning.  
> **username_3**: And you generally want more than one hidden layer, because while a sufficiently large network can capture any function to an arbitrary precision, for certain computations a shallow network would need exponentially more nodes than a one layer deeper network that does the same.  
> **username_2**: @username_3 While I agree that you generally want more than one hidden layer, I wonder if you have a source for the claim that a shallow network needs exponentially more nodes than a one layer deeper network.  
> **username_3**: @MartinThoma https://arxiv.org/pdf/1603.00988.pdf or https://arxiv.org/pdf/1509.08101.pdf are some sources; the particular proofs of course are different for different types of functions; It generally starts with Hastad 1987 "Computational limitations of small-depth circuits" and in general this concept is the counterpoint to the universal approximator theorem - while yes, even the simplest architectures can implement anything, for certain functions they require an unreasonable amount of resources.  
> **username_3**: @MartinThoma http://deeplearning.cs.cmu.edu/pdfs/Delalleau_Bengio_2011.pdf can also be relevant.  

---

### Answer by username_4 on 2016-10-23. Score: 1

Normally one node/layer applies linear fitting of the the input to the hypothesis, in other words uses linear function ($y = ax + b$). Adding layers chains liner functions, potentially allowing fitting higher order functions. A great explanation can be found [here](http://colah.github.io/posts/2015-01-Visualizing-Representations/).


---

