Answers.Body: <p>Backpropagation is actually a lot easier than it is made out to be - <em>if</em> you have a basic understanding of calculus and the chain rule, and the single multi-variable calculus rule that to combine 2 gradient vectors, you simply add them.</p>
  
  <p><a href="https://youtu.be/d14TUNcbn1k?t=354" rel="nofollow noreferrer">This is hands down the best walk through of back prop I've found on the internet.</a> If you are still confused after that, feel free to ask me any further questions. Here is also a quick forward and backward pass example I made for a simple CNN (only a few layers though, and the gradient only goes back to channel 1 of filter 1)<a href="https://i.sstatic.net/hxZBv.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hxZBv.png" alt="backprop"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-09T20:11:57.447Z
Answers.Text: The Stanford university video you liked (ep 4 and 6) are indeed the clearest and simplest explanation on the topic I've found online! Thanks again!
Answers.UserDisplayName: Fabrizio
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-09T01:01:41.583Z
Answers.LastActivityDate: 2020-01-09T01:01:41.583Z
Answers.UserDisplayName: Recessive
Body: <p>I am quite new in the AI field. I am trying to create a neural network, in a language (Dart) where I couldn't find examples or premade libraries or tutorials. I've tried looking online for a strictly "vanilla" python implementation (without third-party libraries), but I couldn't find any. </p>
  
  <p>I've found a single layer implementation, but it's done only with matrices and it's quite cryptic for a beginner.</p>
  
  <p>I've understood the idea between the feed forwarding, a neuron calculates the sum of its inputs, adds a bias and activates it. </p>
  
  <p>But I couldn't find anything a neuron-level explanation of the math behind backpropagation. (By neuron-level I think of the math down to the single neuron as a sequence of operations instead of multiple neurons treated as matrices). </p>
  
  <p>What is the math behind it? Are there any resources to learn it that are suitable as a beginner?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-06T21:46:04.787Z
Comments.Text: backpropagation is a bit harder. You basically calculate a loss function between the values you predict and the labels, and then propagate it with some form of gradient descend to the weights in different layers. Also implementing it without matrices might be very inefficient. If you search online for "backpropagation simple implementation" or "backpropagation math step by step" or something else I'm sure you will find plenty of examples.
Comments.UserDisplayName: Miguel Saraiva
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-06T22:11:13.243Z
Comments.Text: You can think of the matrix math as just a notation for the several individual multiplications, etc. not magic.
Comments.UserDisplayName: George White
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-06T22:59:09.270Z
Comments.Text: Here is an example with values which may help to explain what is happening. https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/
Comments.UserDisplayName: lfgtm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-07T12:49:04.547Z
Comments.Text: Check Michael Nielsen's book [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com/). Math of backpropagation is discussed at length and he does not use any libraries - just Python.
Comments.UserDisplayName: serali
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-08T15:01:07.007Z
Comments.Text: you begin by getting the loss of the output then you propagate and apply gradients back through the layers using the chain rule, to do a few layers of this  statically isnt hard just verbose, to do so for arbitrary numbers of layers means your looking at implementing something close to autodiff
Comments.UserDisplayName: nickw
Tags: <backpropagation><math><resource-request>
Title: What is the neuron-level math behind backpropagation for a neural network?
UserDisplayName: Fabrizio
ThreadText: # What is the neuron-level math behind backpropagation for a neural network?
  
  (Asked by: username_0 on 2020-01-06)
  
  I am quite new in the AI field. I am trying to create a neural network, in a language (Dart) where I couldn't find examples or premade libraries or tutorials. I've tried looking online for a strictly "vanilla" python implementation (without third-party libraries), but I couldn't find any.
  
  I've found a single layer implementation, but it's done only with matrices and it's quite cryptic for a beginner.
  
  I've understood the idea between the feed forwarding, a neuron calculates the sum of its inputs, adds a bias and activates it.
  
  But I couldn't find anything a neuron-level explanation of the math behind backpropagation. (By neuron-level I think of the math down to the single neuron as a sequence of operations instead of multiple neurons treated as matrices).
  
  What is the math behind it? Are there any resources to learn it that are suitable as a beginner?
  
  **Tags:** backpropagation, math, resource-request
  > **Comments:**
  > **username_1**: backpropagation is a bit harder. You basically calculate a loss function between the values you predict and the labels, and then propagate it with some form of gradient descend to the weights in different layers. Also implementing it without matrices might be very inefficient. If you search online for "backpropagation simple implementation" or "backpropagation math step by step" or something else I'm sure you will find plenty of examples.  
  > **username_2**: You can think of the matrix math as just a notation for the several individual multiplications, etc. not magic.  
  > **username_3**: Here is an example with values which may help to explain what is happening. https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/  
  > **username_4**: Check Michael Nielsen's book [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com/). Math of backpropagation is discussed at length and he does not use any libraries - just Python.  
  > **username_5**: you begin by getting the loss of the output then you propagate and apply gradients back through the layers using the chain rule, to do a few layers of this  statically isnt hard just verbose, to do so for arbitrary numbers of layers means your looking at implementing something close to autodiff  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_6 on 2020-01-09. Score: 1 | Accepted answer
  
  Backpropagation is actually a lot easier than it is made out to be - *if* you have a basic understanding of calculus and the chain rule, and the single multi-variable calculus rule that to combine 2 gradient vectors, you simply add them.
  
  [This is hands down the best walk through of back prop I've found on the internet.](https://youtu.be/d14TUNcbn1k?t=354) If you are still confused after that, feel free to ask me any further questions. Here is also a quick forward and backward pass example I made for a simple CNN (only a few layers though, and the gradient only goes back to channel 1 of filter 1)[![backprop](https://i.sstatic.net/hxZBv.png)](https://i.sstatic.net/hxZBv.png)
  
  > **Comments:**
  > **username_0**: The Stanford university video you liked (ep 4 and 6) are indeed the clearest and simplest explanation on the topic I've found online! Thanks again!  
  
  ---
  
  