Answers.Body: <p>I will overly simplify ANNs in order to point how they work. Examples might not be 100% accurate.</p>
  
  <p>In the simplest form, network is trained using the apriori information extracted from the ground truth. This basically means that ANN uses the relation between the input and output. </p>
  
  <p>For instance, if you are to classify shrubs and trees, one of the input could be height and the other could be the width of the tree. Now, if you have only input and output layers, increasing height means increasing chance for the object to be a tree. Thus, input height would have a positive weight connecting to tree output and a negative weight to shrub output. However, as the plant gets wider, the chance of it being a shrub increases. Taller shrubs are wider than shorter ones. Thus input weight would have positive weight connecting to the shrub output. Finally, the chance of being a tree is not affected by the width and thus will have close to 0 weight between this input and output. This network will effectively work like a linear discriminant classifier.</p>
  
  <p>Now instead of assigning weights by hand, you may use a learning algorithm that tries to adjust weights so that the output is correct when the series of input is supplied. Ideally this training algorithm should reach to the conclusion that we have made in the previous example. Most training algorithms are recursive. They supply the inputs multiple times, and in a simple sense, they reward pathways that are correct by increasing their weights and punishes pathways that are causing incorrect answer.</p>
  
  <p>When hidden layers are used in a system, they would be able to correlate input on higher degrees. Thus, as the number of layers get higher, ANN learns the input set much better. However, this does not mean it gets better. If the ANN over fits the input set, it would be affected from the random noise that is in the dataset. This problem is generally referred as memorization. There are learning algorithms that try to minimize memorization and maximize generalization ability. But ultimately, the number of training samples should be high enough so that ANN cannot overfit to the data.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-11T09:25:35.303Z
Answers.Text: You talk about weights being determined by the learning process. How are they used to calculate the value of a node given all its input nodes? What formula is used? What's the math behind all this?
Answers.UserDisplayName: null
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-11T09:41:36.330Z
Answers.Text: I tried to give an abstract view of an ANN. A node's value generally determined sum(input * weight). However, depending on the network there might be some transform function. Training methods vary wildly from method to method and even a single method can be tuned using different parameters. I can name you an easy starting point: feed forward neural networks with backpropagation learning algorithm. There are many sources online will walk through the maths of this method.
Answers.UserDisplayName: Cem Kalyoncu
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-11T09:47:52.883Z
Answers.Text: Thanks for the additional information. I was just looking for an explanation beyond the usually high abstraction explanation.
Answers.UserDisplayName: null
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-11T09:00:25.247Z
Answers.LastActivityDate: 2016-08-11T09:00:25.247Z
Answers.UserDisplayName: Cem Kalyoncu
Answers.Body: <p>I'll try to do something intuitive; Each node in a neural network is referred to as a neuron. To understand what's going on under the hood of a neural network you only really need to understand an individual neuron. </p>
  
  <p>Now each neuron has a set of inputs (other neurons; they can potentially be the inputs to the network as a whole as well), and each input has a weight associated with it. Every time the network is used, each neuron computes its output as the weighted sum of its inputs passed through some gate (The "Activation Function", a mathematical function designed to get a particular behaviour. For example sigmoid AF takes an input of any size and transforms it into an output in the range [0, 1].) Obviously, this is driven from the inputs to the neural network so that no neuron is computing its outputs before all of the neurons used as its inputs have done the same.</p>
  
  <p>When you refer to the value of a node; there isn't a single value. Each neuron has several weights associated with it as it may be the input to several other neurons, and each of those neurons assigns it a different weight. Instead, it is better to thing of a neural network as a directed graph of nodes (neurons) which are labelled with a particular activation function, and edges (input/output connections) which are labelled with a particular weight. While the structure and activation functions used in the neural networks is a matter of topology design, there are a number of algorithms for designing a ANN for a particular topology.</p>
  
  <p>The most commonly used (and possibly easiest to explain) is backpropagation. In pseudo-Layman's terms we start off with random weights on all edges in the network. We then compute the output of the network for a training set (a set of known input/output pairs). By careful choice of activation function, it is possible to differentiate the error (computed analogously to the expected output minus the actual output of the ANN for each input/output pair) with respect to the weights of the neural network. This allows us to compute a gradient for each weight; the direction in which we can move the weight to reduce the error on the training set. By doing this until we find an optima (a point where all movements increase error), we can find some 'good' configuration of weights for that particular ANN. </p>
  
  <p>There's a nice tutorial on BP here <a href="http://www.cse.unsw.edu.au/~cs9417ml/MLP2/BackPropagation.html" rel="nofollow noreferrer">http://www.cse.unsw.edu.au/~cs9417ml/MLP2/BackPropagation.html</a>. The diagram associated with it does nicely to explain my point:
  <a href="https://i.sstatic.net/O2d6L.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/O2d6L.jpg" alt="enter image description here"></a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-18T09:52:04.357Z
Answers.LastActivityDate: 2016-08-18T09:52:04.357Z
Answers.UserDisplayName: Tim Atkinson
Body: <p>When it comes to neural networks, it's often only explained what abstract task they do, say for example detect a number in an image. I never understood what's going on under the hood essentially.</p>
  
  <p>There seems to be a common structure of a directed graph, with values in each node. Some nodes are input nodes. Their values can be set. The values of subsequent nodes are then calculated based on those along the edges of the graph until the values for the output nodes are set, which can be interpreted a result.</p>
  
  <p>How exactly is the value of each node determined? I assume that some formula is associated with each node that takes all incoming nodes as input to calculate the value of the node. What formula is used? Is the formula the same throughout the network?</p>
  
  <p>Then I heard that a network has to be trained. I assume that such training would be the process to assign values to coefficients of the formulas used to determine the node values. Is that correct?</p>
  
  <p>In layman's terms, what are the underlying principles that make a neural network work?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-12T09:46:03.867Z
Comments.Text: This is textbook material; there is little use in reproducing it here.
Comments.UserDisplayName: Raphael
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-12T09:46:20.427Z
Comments.Text: I'm voting to close this question as off-topic because it asks for general reference found in textbooks.
Comments.UserDisplayName: Raphael
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-12T10:12:06.477Z
Comments.Text: Books usually explaining that in difficult and technical ways, this can allow here some creativity.
Comments.UserDisplayName: kenorb
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-12T10:53:34.247Z
Comments.Text: @Raphael same holds true for the author of the second textbook. If redundancy is an issue for you, why did you leave two comments both stating that this information can be found in textbooks?
Comments.UserDisplayName: null
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-12T11:15:43.797Z
Comments.Text: @Gleemax Leaving two comments is just a issue with the system, that leaves an automatic comment when voting to close with a custom reason.
Comments.UserDisplayName: wythagoras
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-12T11:22:13.880Z
Comments.Text: @wythagoras I know. I just couldn't keep myself from pointing out that redundancy.
Comments.UserDisplayName: null
Tags: <neural-networks>
Title: How exactly is the value of each node determined? Is it the same formula throughout the network?
UserDisplayName: null
ThreadText: # How exactly is the value of each node determined? Is it the same formula throughout the network?
  
  (Asked by: username_0 on 2016-08-11)
  
  When it comes to neural networks, it's often only explained what abstract task they do, say for example detect a number in an image. I never understood what's going on under the hood essentially.
  
  There seems to be a common structure of a directed graph, with values in each node. Some nodes are input nodes. Their values can be set. The values of subsequent nodes are then calculated based on those along the edges of the graph until the values for the output nodes are set, which can be interpreted a result.
  
  How exactly is the value of each node determined? I assume that some formula is associated with each node that takes all incoming nodes as input to calculate the value of the node. What formula is used? Is the formula the same throughout the network?
  
  Then I heard that a network has to be trained. I assume that such training would be the process to assign values to coefficients of the formulas used to determine the node values. Is that correct?
  
  In layman's terms, what are the underlying principles that make a neural network work?
  
  **Tags:** neural-networks
  > **Comments:**
  > **username_1**: This is textbook material; there is little use in reproducing it here.  
  > **username_1**: I'm voting to close this question as off-topic because it asks for general reference found in textbooks.  
  > **username_2**: Books usually explaining that in difficult and technical ways, this can allow here some creativity.  
  > **username_0**: @Raphael same holds true for the author of the second textbook. If redundancy is an issue for you, why did you leave two comments both stating that this information can be found in textbooks?  
  > **username_3**: @Gleemax Leaving two comments is just a issue with the system, that leaves an automatic comment when voting to close with a custom reason.  
  > **username_0**: @wythagoras I know. I just couldn't keep myself from pointing out that redundancy.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2016-08-11. Score: 4
  
  I will overly simplify ANNs in order to point how they work. Examples might not be 100% accurate.
  
  In the simplest form, network is trained using the apriori information extracted from the ground truth. This basically means that ANN uses the relation between the input and output.
  
  For instance, if you are to classify shrubs and trees, one of the input could be height and the other could be the width of the tree. Now, if you have only input and output layers, increasing height means increasing chance for the object to be a tree. Thus, input height would have a positive weight connecting to tree output and a negative weight to shrub output. However, as the plant gets wider, the chance of it being a shrub increases. Taller shrubs are wider than shorter ones. Thus input weight would have positive weight connecting to the shrub output. Finally, the chance of being a tree is not affected by the width and thus will have close to 0 weight between this input and output. This network will effectively work like a linear discriminant classifier.
  
  Now instead of assigning weights by hand, you may use a learning algorithm that tries to adjust weights so that the output is correct when the series of input is supplied. Ideally this training algorithm should reach to the conclusion that we have made in the previous example. Most training algorithms are recursive. They supply the inputs multiple times, and in a simple sense, they reward pathways that are correct by increasing their weights and punishes pathways that are causing incorrect answer.
  
  When hidden layers are used in a system, they would be able to correlate input on higher degrees. Thus, as the number of layers get higher, ANN learns the input set much better. However, this does not mean it gets better. If the ANN over fits the input set, it would be affected from the random noise that is in the dataset. This problem is generally referred as memorization. There are learning algorithms that try to minimize memorization and maximize generalization ability. But ultimately, the number of training samples should be high enough so that ANN cannot overfit to the data.
  
  > **Comments:**
  > **username_0**: You talk about weights being determined by the learning process. How are they used to calculate the value of a node given all its input nodes? What formula is used? What's the math behind all this?  
  > **username_4**: I tried to give an abstract view of an ANN. A node's value generally determined sum(input * weight). However, depending on the network there might be some transform function. Training methods vary wildly from method to method and even a single method can be tuned using different parameters. I can name you an easy starting point: feed forward neural networks with backpropagation learning algorithm. There are many sources online will walk through the maths of this method.  
  > **username_0**: Thanks for the additional information. I was just looking for an explanation beyond the usually high abstraction explanation.  
  
  ---
  
  ### Answer by username_5 on 2016-08-18. Score: 2
  
  I'll try to do something intuitive; Each node in a neural network is referred to as a neuron. To understand what's going on under the hood of a neural network you only really need to understand an individual neuron.
  
  Now each neuron has a set of inputs (other neurons; they can potentially be the inputs to the network as a whole as well), and each input has a weight associated with it. Every time the network is used, each neuron computes its output as the weighted sum of its inputs passed through some gate (The "Activation Function", a mathematical function designed to get a particular behaviour. For example sigmoid AF takes an input of any size and transforms it into an output in the range [0, 1].) Obviously, this is driven from the inputs to the neural network so that no neuron is computing its outputs before all of the neurons used as its inputs have done the same.
  
  When you refer to the value of a node; there isn't a single value. Each neuron has several weights associated with it as it may be the input to several other neurons, and each of those neurons assigns it a different weight. Instead, it is better to thing of a neural network as a directed graph of nodes (neurons) which are labelled with a particular activation function, and edges (input/output connections) which are labelled with a particular weight. While the structure and activation functions used in the neural networks is a matter of topology design, there are a number of algorithms for designing a ANN for a particular topology.
  
  The most commonly used (and possibly easiest to explain) is backpropagation. In pseudo-Layman's terms we start off with random weights on all edges in the network. We then compute the output of the network for a training set (a set of known input/output pairs). By careful choice of activation function, it is possible to differentiate the error (computed analogously to the expected output minus the actual output of the ANN for each input/output pair) with respect to the weights of the neural network. This allows us to compute a gradient for each weight; the direction in which we can move the weight to reduce the error on the training set. By doing this until we find an optima (a point where all movements increase error), we can find some 'good' configuration of weights for that particular ANN.
  
  There's a nice tutorial on BP here <http://www.cse.unsw.edu.au/~cs9417ml/MLP2/BackPropagation.html>. The diagram associated with it does nicely to explain my point:
  [![enter image description here](https://i.sstatic.net/O2d6L.jpg)](https://i.sstatic.net/O2d6L.jpg)
  
  
  ---
  
  