Answers.Body: <p>Hidden layers by themselves aren't useful. If you had hidden layers that were linear, the end result would still be a linear function of the inputs, and so you could collapse an arbitrary number of linear layers down to a single layer.</p>
  
  <p>This is why we use nonlinear <a href="https://en.wikipedia.org/wiki/Activation_function" rel="nofollow">activation functions</a>, like RELU. This allows us to add a level of nonlinear complexity with each hidden layer, and with arbitrarily many hidden layers we can construct arbitrarily complicated nonlinear functions.</p>
  
  <p>Because we can (at least in theory) capture any degree of complexity, we think of neural networks as "universal learners," in that a large enough network could mimic any function.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-02T16:15:49.970Z
Answers.LastActivityDate: 2016-08-02T16:15:49.970Z
Answers.UserDisplayName: Matthew Gray
Answers.Body: <p>Actually, the hierarchical learning explanation given by <a href="https://ai.stackexchange.com/a/51/2444">mindcrime</a> is not that acceptable anymore (<a href="https://youtu.be/Z6rxFNMGdn0?t=260" rel="nofollow noreferrer">This</a> was also indicated by Ian Goodfellow). Since there are neural networks with 150 layers or more, and this explanation does not make sense for such neural networks.  However, we can think of it as solving the knots of high dimensional manifolds, i.e. we transform the input into high dimensional space, and this helps us to find a better representation of the data.</p>
  <p>A geometric interpretation was explained as such in the book <em>Deep Learning with Python</em> by François Chollet:</p>
  <blockquote>
  <p>...you can interpret a neural network as a very complex geometric transformation in a high-dimensional space, implemented via a long series of simple steps...</p>
  </blockquote>
  <blockquote>
  <p>Imagine two sheets of colored paper: one red and one blue. Put one on top of the other. Now crumple them together into a small ball. That crumpled paper ball is your input data, and each sheet of paper is a class of data in a classification problem. What a neural network (or any other machine-learning model) is meant to do is figure out a transformation of the paper ball that would uncrumple it, so as to make the two classes cleanly separable again. With deep learning, this would be implemented as a series of simple transformations of the 3D space, such as those you could apply on the paper ball with your fingers, one movement at a time. Uncrumpling paper balls is what machine learning is about: finding neat representations for complex, highly folded data manifolds. At this point, you should have a pretty good intuition as to why deep learning excels at this: it takes the approach of incrementally decomposing a complicated geometric transformation into a long chain of elementary ones, which is pretty much the strategy a human would follow to uncrumple a paper ball. Each layer in a deep network applies a transformation that disentangles the data a little—and a deep stack of layers makes tractable an extremely complicated disentanglement process.</p>
  </blockquote>
  <p>I suggest you to read <a href="https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/" rel="nofollow noreferrer">this brilliant blog post</a> to learn about the topological interpretation of deep learning.</p>
  <p>Also, <a href="https://cs.stanford.edu/people/karpathy/convnetjs/demo/classify2d.html" rel="nofollow noreferrer">this</a> toy interactive code may help you.</p>
  <p>In the context of machine learning, the concept of a <em>manifold</em> can be illustrated as in the following figure.</p>
  <p><a href="https://i.sstatic.net/bpS1y.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bpS1y.png" alt="swiss" /></a></p>
  <p>In the first part, data are 3-dimensional. However, we can find a transformation to get the second image, which shows that data is actually artificially high dimensional, i.e. it is a 2-dimensional manifold in 3-D space. This example may be thought of as a classification problem, and colors may represent classes, and we can find a trivial representation of the data for classification.</p>
  <p>Another example could be following figures from the blog I mentioned. In here, this classification problem cannot be solved without having a layer that has 3 or more hidden units, regardless of depth. So the notion of high dimensional transformation is important.</p>
  <p><a href="https://i.sstatic.net/NfkSD.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/NfkSD.jpg" alt="1st image" /></a></p>
  <p>We can map this data to 3-D, and find a plane to separate them.</p>
  <p><a href="https://i.sstatic.net/tyRcb.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/tyRcb.jpg" alt="2nd iamge" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-10T13:29:49.417Z
Answers.LastActivityDate: 2021-01-10T15:08:04.167Z
Answers.LastEditDate: 2021-01-10T15:08:04.167Z
Answers.UserDisplayName: Enes
Answers.Body: <p>One aspect that I'd like to add to the previous answers is the so-called <em>Curse of dimensionality</em>. This concept refers to the problem that many algorithms have a time complexity that grows exponentially with the dimension of the data.</p>
  <p>As a simple example, let us consider a set <span class="math-container">$\{0,1\}^{D}$</span> that has only two values per dimension. For example, <span class="math-container">$\{0,1\}^{2} = \{(0,0),(0,1),(1,0),(1,1)\}$</span> and <span class="math-container">$(0,1,0) \in \{0,1\}^{3}$</span>. Now imagine that you are given a function <span class="math-container">$f: \{0,1\}^{\times D} \rightarrow \{TRUE, FALSE\}$</span> that outputs TRUE exactly for one particular input. The goal is to determine that input.</p>
  <p>In the example, if nothing else is known about f, the best thing one can do is to try the inputs one after another. However, <span class="math-container">$\{0,1\}^{D}$</span> has <span class="math-container">$2^D$</span> elements. So the number of inputs one has to try out will in general be roughly <span class="math-container">$2^D$</span> as well.</p>
  <p>However, there exist examples suffering from a curse of dimensionality that can be solved with deep learning, i.e. using neural networks with <strong>many</strong> hidden layers.
  One example of great practical importance is given by high-dimensional partial differential equations, see e.g this report:</p>
  <p><a href="http://www.sam.math.ethz.ch/sam_reports/reports_final/reports2017/2017-44_fp.pdf" rel="nofollow noreferrer">http://www.sam.math.ethz.ch/sam_reports/reports_final/reports2017/2017-44_fp.pdf</a></p>
  <p>or this example for heat equations:</p>
  <p><a href="https://arxiv.org/abs/1901.10854" rel="nofollow noreferrer">https://arxiv.org/abs/1901.10854</a></p>
  <p>I also found this review on using deep learning to overcome the curse of dimensionality:</p>
  <p><a href="https://cbmm.mit.edu/sites/default/files/publications/02_761-774_00966_Bpast.No_.66-6_28.12.18_K1.pdf" rel="nofollow noreferrer">https://cbmm.mit.edu/sites/default/files/publications/02_761-774_00966_Bpast.No_.66-6_28.12.18_K1.pdf</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-10T23:11:28.690Z
Answers.Text: Hi. Welcome to AI SE and thanks for contributing to our site. The "curse of dimensionality" problem is an important problem in AI. However, this answer does not really explain **why** (multiple) hidden layers solve that issue. So, I suggest that you provide more details about that (maybe after reading those or other papers, if you haven't).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-10T22:56:08.420Z
Answers.LastActivityDate: 2021-01-10T22:56:08.420Z
Answers.UserDisplayName: QuantumAI
Answers.Body: <p>"Hidden" layers really aren't all that special... a hidden layer is really no more than any layer that isn't input or output. So even a very simple 3 layer NN has 1 hidden layer. So I think the question isn't really "How do hidden layers help?" as much as "Why are deeper networks better?".  </p>
  
  <p>And the answer to that latter question is an area of active research. Even top experts like Geoffrey Hinton and Andrew Ng will freely admit that we don't really understand why deep neural networks work. That is, we don't understand them in complete detail anyway.</p>
  
  <p>That said, the theory, as I understand it goes something like this...  successive layers of the network learn successively more sophisticated features, which build on the features from preceding layers. So, for example, an NN used for facial recognition might work like this: the first layer detects edges and nothing else. The next layer up recognizes geometric shapes (boxes, circles, etc.). The next layer up recognizes primitive features of a face, like eyes, noses, jaw, etc. The next layer up then recognizes composites based on combinations of "eye" features, "nose" features, and so on.  </p>
  
  <p>So, in theory, deeper networks (more hidden layers) are better in that they develop a more granular/detailed representation of a "thing" being recognized.  </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-24T16:48:01.467Z
Answers.Text: One hidden layer with non-linearities allows a NN to approximate any continuous function. A perceptron cannot do that. So, a hidden layer **can** actually be special.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-02T16:16:59.330Z
Answers.LastActivityDate: 2018-04-12T02:30:32.837Z
Answers.LastEditDate: 2018-04-12T02:30:32.837Z
Answers.UserDisplayName: mindcrime
Body: <p>Why would anybody want to use "hidden layers"? How do they enhance the learning ability of the network in comparison to the network which doesn't have them (linear models)?</p>
  
Tags: <neural-networks><deep-learning><deep-neural-networks><hidden-layers>
Title: What is the purpose of the hidden layers?
UserDisplayName: kenorb
ThreadText: # What is the purpose of the hidden layers?
  
  (Asked by: username_0 on 2016-08-02)
  
  Why would anybody want to use "hidden layers"? How do they enhance the learning ability of the network in comparison to the network which doesn't have them (linear models)?
  
  **Tags:** neural-networks, deep-learning, deep-neural-networks, hidden-layers
  
  ---
  
  ## 4 answers
  
  ### Answer by username_5 on 2016-08-02. Score: 5 | Accepted answer
  
  "Hidden" layers really aren't all that special... a hidden layer is really no more than any layer that isn't input or output. So even a very simple 3 layer NN has 1 hidden layer. So I think the question isn't really "How do hidden layers help?" as much as "Why are deeper networks better?".
  
  And the answer to that latter question is an area of active research. Even top experts like Geoffrey Hinton and Andrew Ng will freely admit that we don't really understand why deep neural networks work. That is, we don't understand them in complete detail anyway.
  
  That said, the theory, as I understand it goes something like this... successive layers of the network learn successively more sophisticated features, which build on the features from preceding layers. So, for example, an NN used for facial recognition might work like this: the first layer detects edges and nothing else. The next layer up recognizes geometric shapes (boxes, circles, etc.). The next layer up recognizes primitive features of a face, like eyes, noses, jaw, etc. The next layer up then recognizes composites based on combinations of "eye" features, "nose" features, and so on.
  
  So, in theory, deeper networks (more hidden layers) are better in that they develop a more granular/detailed representation of a "thing" being recognized.
  
  > **Comments:**
  > **username_4**: One hidden layer with non-linearities allows a NN to approximate any continuous function. A perceptron cannot do that. So, a hidden layer **can** actually be special.  
  
  ---
  
  ### Answer by username_1 on 2016-08-02. Score: 3
  
  Hidden layers by themselves aren't useful. If you had hidden layers that were linear, the end result would still be a linear function of the inputs, and so you could collapse an arbitrary number of linear layers down to a single layer.
  
  This is why we use nonlinear [activation functions](https://en.wikipedia.org/wiki/Activation_function), like RELU. This allows us to add a level of nonlinear complexity with each hidden layer, and with arbitrarily many hidden layers we can construct arbitrarily complicated nonlinear functions.
  
  Because we can (at least in theory) capture any degree of complexity, we think of neural networks as "universal learners," in that a large enough network could mimic any function.
  
  
  ---
  
  ### Answer by username_2 on 2021-01-10. Score: 2
  
  Actually, the hierarchical learning explanation given by [mindcrime](https://ai.stackexchange.com/a/51/2444) is not that acceptable anymore ([This](https://youtu.be/Z6rxFNMGdn0?t=260) was also indicated by Ian Goodfellow). Since there are neural networks with 150 layers or more, and this explanation does not make sense for such neural networks. However, we can think of it as solving the knots of high dimensional manifolds, i.e. we transform the input into high dimensional space, and this helps us to find a better representation of the data.
  
  A geometric interpretation was explained as such in the book *Deep Learning with Python* by François Chollet:
  
  > ...you can interpret a neural network as a very complex geometric transformation in a high-dimensional space, implemented via a long series of simple steps...
  
  > Imagine two sheets of colored paper: one red and one blue. Put one on top of the other. Now crumple them together into a small ball. That crumpled paper ball is your input data, and each sheet of paper is a class of data in a classification problem. What a neural network (or any other machine-learning model) is meant to do is figure out a transformation of the paper ball that would uncrumple it, so as to make the two classes cleanly separable again. With deep learning, this would be implemented as a series of simple transformations of the 3D space, such as those you could apply on the paper ball with your fingers, one movement at a time. Uncrumpling paper balls is what machine learning is about: finding neat representations for complex, highly folded data manifolds. At this point, you should have a pretty good intuition as to why deep learning excels at this: it takes the approach of incrementally decomposing a complicated geometric transformation into a long chain of elementary ones, which is pretty much the strategy a human would follow to uncrumple a paper ball. Each layer in a deep network applies a transformation that disentangles the data a little—and a deep stack of layers makes tractable an extremely complicated disentanglement process.
  
  I suggest you to read [this brilliant blog post](https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/) to learn about the topological interpretation of deep learning.
  
  Also, [this](https://cs.stanford.edu/people/karpathy/convnetjs/demo/classify2d.html) toy interactive code may help you.
  
  In the context of machine learning, the concept of a *manifold* can be illustrated as in the following figure.
  
  [![swiss](https://i.sstatic.net/bpS1y.png)](https://i.sstatic.net/bpS1y.png)
  
  In the first part, data are 3-dimensional. However, we can find a transformation to get the second image, which shows that data is actually artificially high dimensional, i.e. it is a 2-dimensional manifold in 3-D space. This example may be thought of as a classification problem, and colors may represent classes, and we can find a trivial representation of the data for classification.
  
  Another example could be following figures from the blog I mentioned. In here, this classification problem cannot be solved without having a layer that has 3 or more hidden units, regardless of depth. So the notion of high dimensional transformation is important.
  
  [![1st image](https://i.sstatic.net/NfkSD.jpg)](https://i.sstatic.net/NfkSD.jpg)
  
  We can map this data to 3-D, and find a plane to separate them.
  
  [![2nd iamge](https://i.sstatic.net/tyRcb.jpg)](https://i.sstatic.net/tyRcb.jpg)
  
  
  ---
  
  ### Answer by username_3 on 2021-01-10. Score: 0
  
  One aspect that I'd like to add to the previous answers is the so-called *Curse of dimensionality*. This concept refers to the problem that many algorithms have a time complexity that grows exponentially with the dimension of the data.
  
  As a simple example, let us consider a set $\{0,1\}^{D}$ that has only two values per dimension. For example, $\{0,1\}^{2} = \{(0,0),(0,1),(1,0),(1,1)\}$ and $(0,1,0) \in \{0,1\}^{3}$. Now imagine that you are given a function $f: \{0,1\}^{\times D} \rightarrow \{TRUE, FALSE\}$ that outputs TRUE exactly for one particular input. The goal is to determine that input.
  
  In the example, if nothing else is known about f, the best thing one can do is to try the inputs one after another. However, $\{0,1\}^{D}$ has $2^D$ elements. So the number of inputs one has to try out will in general be roughly $2^D$ as well.
  
  However, there exist examples suffering from a curse of dimensionality that can be solved with deep learning, i.e. using neural networks with **many** hidden layers.
  One example of great practical importance is given by high-dimensional partial differential equations, see e.g this report:
  
  <http://www.sam.math.ethz.ch/sam_reports/reports_final/reports2017/2017-44_fp.pdf>
  
  or this example for heat equations:
  
  <https://arxiv.org/abs/1901.10854>
  
  I also found this review on using deep learning to overcome the curse of dimensionality:
  
  <https://cbmm.mit.edu/sites/default/files/publications/02_761-774_00966_Bpast.No_.66-6_28.12.18_K1.pdf>
  
  > **Comments:**
  > **username_4**: Hi. Welcome to AI SE and thanks for contributing to our site. The "curse of dimensionality" problem is an important problem in AI. However, this answer does not really explain **why** (multiple) hidden layers solve that issue. So, I suggest that you provide more details about that (maybe after reading those or other papers, if you haven't).  
  
  ---
  
  