Answers.Body: <p>When you are training a neural network, you use an algorithm called back propagation. This algorithm uses partial derivatives to determine the optimal values for weights. Partial derivatives are a calculus based method which tell you how far you need to adjust the weights in order to get to an optimum value. However, when you have neural networks with many different weights you need to be able to decide which ones to update in order to achieve a more optimal network. This is where stochastic gradient descent comes in. This algorithm is able to take many different gradients and traverse through many dimensional space into a lower error rate. Also, from your diagram of your neural network, I do not see any hidden neurons, which are neurons between the input and output layers which do most of the actual work. While you might be able to train your neural network optimally without hidden neurons, it might improve your accuracy depending upon your dataset. Finally, implementing a back propagation algorithm is a lot of work which is why many people use pre built libraries. Unless you are doing research, I recommend you do the same. I think for a problem like this, pybrain would be perfect, and if you want to use deep neural networks, tensorflow would be good for that.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-06T15:11:07.480Z
Answers.Text: Thanks a lot! I didnt include hidden neurons because I didn't understand them well. From what I know so far is that they do some math function to the input, which for me is done on the lines (like a function diagram). I wish to understand the fundamentals of neural networks before I approach a library too. The only machine learning code that I have up and running is one that predicts the sum of two numbers.
Answers.UserDisplayName: See Jian Shin
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-06T15:22:33.650Z
Answers.Text: Update : my code somehow works on multiplication, division and subtraction too, and I have no idea why but am very very happy about it.
Answers.UserDisplayName: See Jian Shin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-14T01:04:51.013Z
Answers.Text: See Jian Shin, a hidden layer is a set of neurons just like the input neurons and the output neurons. The hidden layer is the part you actually train in machine learning. All neurons represent a math equation, usually it takes in all of the ouputs from the previous layer, multiplies them by a weight matrix, adds a bias matrix, sums them all up, and applies an activation function used to frame the data for the output.
Answers.UserDisplayName: Jordan LaPrise
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-06T14:42:03.267Z
Answers.LastActivityDate: 2017-10-06T14:42:03.267Z
Answers.UserDisplayName: Aiden Grossman
Body: <p>I recently started working on very simple machine learning codes in Python and I came across a big problem: teaching the system to improve on its guesses.</p>
  
  <p>So this is what the code is about: 
  I will have a list of organisms with their features stated in numerical values. I want to write a code that identifies that whether the organism is a cat or a fish or neither based on their characteristics. (For example an organism with a high fur value and 4 legs is more probable to be a cat.)</p>
  
  <p>My idea for the neural network is to have five input nodes(for the five characteristics) and 2 output nodes (one for how cat it is and one for how fish it is). The input nodes are multiplied by a weight value and then all summed together to produce one of the output nodes. This repeats itself for the other output. How much the system got wrong is just the difference between the value of the output nodes and how cat/fish the being actually is. </p>
  
  <p>But how can I use this information to correct the weights of the input nodes? Since the weights are randomly generated they can be in the "wrong" directions to begin with. For example if the subject is a cat then we should be expecting high fur and leg values. But what if the weight for the leg value is negative while the weight for the fur value is positive? Adding or multiplying the weight by the error won't bring us any closer to accurately determining the being. Is my neural network flawed to begin with? Or is there a rule of thumb in choosing back propagation algorithms?</p>
  
  <p>Thanks.
  <a href="https://i.sstatic.net/0LDQ0.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0LDQ0.jpg" alt="the neural network"></a></p>
  
Tags: <machine-learning><backpropagation>
Title: Finding an optimum back propagation algorithm
UserDisplayName: See Jian Shin
ThreadText: # Finding an optimum back propagation algorithm
  
  (Asked by: username_0 on 2017-10-06)
  
  I recently started working on very simple machine learning codes in Python and I came across a big problem: teaching the system to improve on its guesses.
  
  So this is what the code is about:
  I will have a list of organisms with their features stated in numerical values. I want to write a code that identifies that whether the organism is a cat or a fish or neither based on their characteristics. (For example an organism with a high fur value and 4 legs is more probable to be a cat.)
  
  My idea for the neural network is to have five input nodes(for the five characteristics) and 2 output nodes (one for how cat it is and one for how fish it is). The input nodes are multiplied by a weight value and then all summed together to produce one of the output nodes. This repeats itself for the other output. How much the system got wrong is just the difference between the value of the output nodes and how cat/fish the being actually is.
  
  But how can I use this information to correct the weights of the input nodes? Since the weights are randomly generated they can be in the "wrong" directions to begin with. For example if the subject is a cat then we should be expecting high fur and leg values. But what if the weight for the leg value is negative while the weight for the fur value is positive? Adding or multiplying the weight by the error won't bring us any closer to accurately determining the being. Is my neural network flawed to begin with? Or is there a rule of thumb in choosing back propagation algorithms?
  
  Thanks.
  [![the neural network](https://i.sstatic.net/0LDQ0.jpg)](https://i.sstatic.net/0LDQ0.jpg)
  
  **Tags:** machine-learning, backpropagation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2017-10-06. Score: 6 | Accepted answer
  
  When you are training a neural network, you use an algorithm called back propagation. This algorithm uses partial derivatives to determine the optimal values for weights. Partial derivatives are a calculus based method which tell you how far you need to adjust the weights in order to get to an optimum value. However, when you have neural networks with many different weights you need to be able to decide which ones to update in order to achieve a more optimal network. This is where stochastic gradient descent comes in. This algorithm is able to take many different gradients and traverse through many dimensional space into a lower error rate. Also, from your diagram of your neural network, I do not see any hidden neurons, which are neurons between the input and output layers which do most of the actual work. While you might be able to train your neural network optimally without hidden neurons, it might improve your accuracy depending upon your dataset. Finally, implementing a back propagation algorithm is a lot of work which is why many people use pre built libraries. Unless you are doing research, I recommend you do the same. I think for a problem like this, pybrain would be perfect, and if you want to use deep neural networks, tensorflow would be good for that.
  
  > **Comments:**
  > **username_0**: Thanks a lot! I didnt include hidden neurons because I didn't understand them well. From what I know so far is that they do some math function to the input, which for me is done on the lines (like a function diagram). I wish to understand the fundamentals of neural networks before I approach a library too. The only machine learning code that I have up and running is one that predicts the sum of two numbers.  
  > **username_0**: Update : my code somehow works on multiplication, division and subtraction too, and I have no idea why but am very very happy about it.  
  > **username_2**: username_0, a hidden layer is a set of neurons just like the input neurons and the output neurons. The hidden layer is the part you actually train in machine learning. All neurons represent a math equation, usually it takes in all of the ouputs from the previous layer, multiplies them by a weight matrix, adds a bias matrix, sums them all up, and applies an activation function used to frame the data for the output.  
  
  ---
  
  