Answers.Body: <p>I came across the article <a href="http://neuralnetworksanddeeplearning.com/chap4.html" rel="nofollow noreferrer">A visual proof that neural nets can compute any function</a> that explains how a neural network can approximate any function, which pretty much answers my question.</p>
  
  <p>I suppose connections between neurons do not generally arrange themselves in the way described, though, so it would still be interesting to get insight into whether there are patterns of how the inter-neuron connections arrange themselves in neural networks trained to solve actual problems.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-05-30T09:11:30.350Z
Answers.LastActivityDate: 2020-03-07T22:52:46.473Z
Answers.LastEditDate: 2020-03-07T22:52:46.473Z
Answers.UserDisplayName: PeterW
Body: <p>I'm trying to learn about neural networks, and I'm interested in gaining a better conceptual understanding of how they work to solve certain problems. I'm having trouble in conceptually understanding how they succeed in doing regression (i.e. predicting continuous variables), however, and wondered if anybody has a good explanation. I know the mathematics of how NNs work, but a clearer conceptual understanding would be helpful.</p>
  
  <p>To give an idea of what I mean by a "conceptual understanding", here's the one that I have for how multilayer NN's with a sigmoid activation function are able to be effective classifiers. Each NN takes the scalar product of its inputs and a set of weights. The weights define a plane in the input space, and the sign of the scalar product indicates which side of the plane the point defined by the inputs is on. The sigmoid activation function outputs 1 if the point is on one side and 0 or -1 (depending on which function is used) if the point is on the other. So the first hidden layer of neurons can be considered to identify which side of each of a group of planes the input point is on. The neurons can also act as AND and OR gates, so subsequent layers of neurons give an output indicating whether the point lies in a region bounded by several of the planes (e.g. a neuron activates only if the point is above one plane and below another, indicating it is in a region of the space associated with one class of points). So if the network learns an appropriate set of planes to bound regions containing different classes of inputs and the AND/OR relations that determine whether a point is in a given region, then it can classify the input points, and this can work for regions with arbitrarily shaped boundaries.</p>
  
  <p>I've not found or been able to think of a similar way of explaining why an NN can perform well in general regression problems (if it's big enough). Does anyone here know a way to explain this?</p>
  
Tags: <neural-networks><deep-learning>
Title: How do neural networks manage to do regression?
UserDisplayName: PeterW
ThreadText: # How do neural networks manage to do regression?
  
  (Asked by: username_0 on 2017-05-09)
  
  I'm trying to learn about neural networks, and I'm interested in gaining a better conceptual understanding of how they work to solve certain problems. I'm having trouble in conceptually understanding how they succeed in doing regression (i.e. predicting continuous variables), however, and wondered if anybody has a good explanation. I know the mathematics of how NNs work, but a clearer conceptual understanding would be helpful.
  
  To give an idea of what I mean by a "conceptual understanding", here's the one that I have for how multilayer NN's with a sigmoid activation function are able to be effective classifiers. Each NN takes the scalar product of its inputs and a set of weights. The weights define a plane in the input space, and the sign of the scalar product indicates which side of the plane the point defined by the inputs is on. The sigmoid activation function outputs 1 if the point is on one side and 0 or -1 (depending on which function is used) if the point is on the other. So the first hidden layer of neurons can be considered to identify which side of each of a group of planes the input point is on. The neurons can also act as AND and OR gates, so subsequent layers of neurons give an output indicating whether the point lies in a region bounded by several of the planes (e.g. a neuron activates only if the point is above one plane and below another, indicating it is in a region of the space associated with one class of points). So if the network learns an appropriate set of planes to bound regions containing different classes of inputs and the AND/OR relations that determine whether a point is in a given region, then it can classify the input points, and this can work for regions with arbitrarily shaped boundaries.
  
  I've not found or been able to think of a similar way of explaining why an NN can perform well in general regression problems (if it's big enough). Does anyone here know a way to explain this?
  
  **Tags:** neural-networks, deep-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2017-05-30. Score: 8 | Accepted answer
  
  I came across the article [A visual proof that neural nets can compute any function](http://neuralnetworksanddeeplearning.com/chap4.html) that explains how a neural network can approximate any function, which pretty much answers my question.
  
  I suppose connections between neurons do not generally arrange themselves in the way described, though, so it would still be interesting to get insight into whether there are patterns of how the inter-neuron connections arrange themselves in neural networks trained to solve actual problems.
  
  
  ---
  
  