Answers.Body: <p>The key idea of <a href="https://en.wikipedia.org/wiki/Backpropagation" rel="nofollow noreferrer">backpropagation</a> can be summarized as:</p>
  <blockquote>
  <p>Each individual component of the gradient, <span class="math-container">$\partial C/\partial w_{jk}^{l}$</span> can be computed by the chain rule; but doing this separately for each weight is inefficient. Backpropagation efficiently computes the gradient by avoiding duplicate calculations and not computing unnecessary intermediate values, by computing the gradient of each layer – specifically the gradient of the weighted input of each layer, denoted by <span class="math-container">$\delta^{l}$</span> – from back to front... The <span class="math-container">$\delta ^{l}$</span> can easily be computed recursively, going from right to left, as: <span class="math-container">$$\delta ^{l-1}:=(f^{l-1})'\circ (W^{l})^{T}\cdot \delta ^{l}$$</span>
  The gradients of the weights can thus be computed using a few matrix multiplications for each level; this is backpropagation.</p>
  </blockquote>
  <p>Once this crucial step is done, all the weights can be straightforwardly updated via the delta rule involving a learning rate. Therefore you 1st article's quoted section/code just essentially implements the above crucial gradients <span class="math-container">$\delta ^{l-1}$</span> of the weighted inputs of each layer from code line 99-111, and code line 113-125 implements the final weights delta update rule which additionally involves the multiplicative terms of learning rate and the respective weighted inputs of each layer in Numpy's matrix/array linear algebra way.</p>
  <p>And the error signal of your 2nd article's quoted section (&quot;the error signal for neurons in the hidden layer is accumulated from neurons in the output layer&quot;) is just the inner function part <span class="math-container">$(W^{l})^{T}\cdot \delta ^{l}$</span> of the composed gradient function <span class="math-container">$\delta ^{l-1}$</span>. And at the end line 18 of the quoted code snippet you have:</p>
  <p><code>neuron['delta'] = errors[j] * transfer_derivative(neuron['output'])</code></p>
  <p>Thus the same gradient of each layer <span class="math-container">$\delta ^{l-1}$</span> as expressed as <code>neuron['delta']</code> here is obtained too after multiplied by the derivative of the activation/transfer function though here seems only primitive Python data types are employed instead of Numpy's linear algebra data types.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-22T11:05:27.970Z
Answers.Text: Thanks for explaining!  With your explanation, both articles seem similar.  So this means that to calculate the delta of a neuron in the hidden layers, in both ways, both the first article (which uses the current layer) and the second article (which uses the next layer), work equivalently and are based on the same principles for the same goal?, even using a slightly different way? Did I understand correctly?
Answers.UserDisplayName: will The J
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-22T16:09:56.697Z
Answers.Text: Yes, your conclusion is correct, there’s only one standard back propagation algorithm though there’re many different implementations such as manifested here and in another similar question you asked recently. One idea with many different implementations is a common phenomenon in applied sciences and engineering.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-23T01:34:53.470Z
Answers.Text: Thanks!. But, in second article, the sum of DELTA OUTPUT * WEIGHT to calcule delta in hidden layer, he explains that WEIGHT is the weight J (the index of the neuron J of current hidden layer in vector of the output layer, referring to input J), that is, the connection weight of output layer, that is multiplied by input of output layer (the input of the output layer is the output of the last hidden layer). But, in first article is different, because the weights are the weights that connect the input layer input with the current hidden layer. I had this question below:
Answers.UserDisplayName: will The J
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-23T01:35:52.097Z
Answers.Text: I have this another question about the first article: the first article multiplies the delta of the output layer with the weight matrix of the self hidden layer(that is, the weight matrix of the current hidden layer L itself). In contrast, the second article, when calculating the delta in the hidden layer, using a sum, it multiplies the delta of the output layer itself with the weights of the neurons in the output layer itself(Formula sum of DELTA OUPUT * WEIGHT OUPUT). It gets little confusing, and seems to be different calculation. Please, could help me clarify this point?
Answers.UserDisplayName: will The J
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-23T07:03:56.417Z
Answers.Text: The 1st article is using matrix algebra just as my above answer for the hidden layer's all units' gradients $\delta^{l-1}$ of their respective weighted input, so if 3 units in the current hidden layer $\delta^{l-1}$ is a vector of 3 gradients. This [ref](https://home.agh.edu.pl/%7Evlsi/AI/backp_t_en/backprop.html) is clearer, the first hidden layer has 3 units (1,2,3), each has gradient of its own weighted input, and it's not in matrix form, so like your 2nd article, you can compare and check its for-loops carefully though this is mainly a coding practice, I'd prefer 1st article's matrix code.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-23T22:41:43.347Z
Answers.Text: Thank for the explanations!. It has helped me a lot to understand the concepts.
Answers.UserDisplayName: will The J
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-22T07:50:51.210Z
Answers.LastActivityDate: 2023-11-22T07:50:51.210Z
Answers.UserDisplayName: cinch
Body: <p><strong>Article 1:</strong>
  <a href="https://pyimagesearch.com/2021/05/06/backpropagation-from-scratch-with-python/" rel="nofollow noreferrer">https://pyimagesearch.com/2021/05/06/backpropagation-from-scratch-with-python/</a></p>
  <p><strong>Article 2:</strong>
  <a href="https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/" rel="nofollow noreferrer">https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/</a></p>
  <p>I was reading the 2 articles that teach backpropagation from scratch.
  I had a doubt about the first article, and so I would really like to ask specifically about the way he used to calculate the deltas of neurons in the hidden layer.</p>
  <p>The first article, talking about calculating the delta in hidden layers, mentions the following:</p>
  <blockquote>
  <p>the delta of the current layer is equal to the delta from the previous
  layer dotted with the weight matrix of the current layer, followed by
  multiplication of the delta by the derivative of the nonlinear
  activation function for the activations of the current layer</p>
  </blockquote>
  <p>However, in the second article, the explanation seems to be a little different, in the second article's explanation on how to calculate the delta he mentions</p>
  <blockquote>
  <p>You can see that the error signal for neurons in the hidden layer is
  accumulated from neurons in the output layer where the hidden neuron
  number j is also the index of the neuron’s weight in the output layer
  neuron[‘weights’][j].</p>
  </blockquote>
  <p>Analyzing this explanation from the second article, <strong>I had this doubt: the first article on the hidden layer says that the delta of the current layer is equal to the delta from the previous layer dotted with the weight matrix of the current layer. However, in the second article he says that the delta in the hidden layer is calculated using the sum of the deltas of the next layer multiplied by the connection weights with the neuron with index J in the current hidden layer</strong> (in this case, the next layer is the output layer), this sum is multiplied by the derivative of the activation of the hidden layer neuron. I was very confused precisely because in the second article the weights that were used to calculate the delta are not the weights of the current layer (as is done in the first article), instead, the weights used in the second article are the weights of the next layer (That's it, the weights of the inputs of the neurons in the next layer, in this case the output layer)</p>
  <p><strong>I know that both articles are similar explanations, but I was confused. Is there any difference in the formulas of both articles? Or in practice are both equivalent? Could someone please help me understand these differences?</strong></p>
  
Tags: <neural-networks><backpropagation>
Title: Why in one article to calcule the delta uses the weights of current layer, while in another article uses the weights of neuron inputs of next layer?
UserDisplayName: will The J
ThreadText: # Why in one article to calcule the delta uses the weights of current layer, while in another article uses the weights of neuron inputs of next layer?
  
  (Asked by: username_0 on 2023-11-21)
  
  **Article 1:**
  <https://pyimagesearch.com/2021/05/06/backpropagation-from-scratch-with-python/>
  
  **Article 2:**
  <https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/>
  
  I was reading the 2 articles that teach backpropagation from scratch.
  I had a doubt about the first article, and so I would really like to ask specifically about the way he used to calculate the deltas of neurons in the hidden layer.
  
  The first article, talking about calculating the delta in hidden layers, mentions the following:
  
  > the delta of the current layer is equal to the delta from the previous
  > layer dotted with the weight matrix of the current layer, followed by
  > multiplication of the delta by the derivative of the nonlinear
  > activation function for the activations of the current layer
  
  However, in the second article, the explanation seems to be a little different, in the second article's explanation on how to calculate the delta he mentions
  
  > You can see that the error signal for neurons in the hidden layer is
  > accumulated from neurons in the output layer where the hidden neuron
  > number j is also the index of the neuron’s weight in the output layer
  > neuron[‘weights’][j].
  
  Analyzing this explanation from the second article, **I had this doubt: the first article on the hidden layer says that the delta of the current layer is equal to the delta from the previous layer dotted with the weight matrix of the current layer. However, in the second article he says that the delta in the hidden layer is calculated using the sum of the deltas of the next layer multiplied by the connection weights with the neuron with index J in the current hidden layer** (in this case, the next layer is the output layer), this sum is multiplied by the derivative of the activation of the hidden layer neuron. I was very confused precisely because in the second article the weights that were used to calculate the delta are not the weights of the current layer (as is done in the first article), instead, the weights used in the second article are the weights of the next layer (That's it, the weights of the inputs of the neurons in the next layer, in this case the output layer)
  
  **I know that both articles are similar explanations, but I was confused. Is there any difference in the formulas of both articles? Or in practice are both equivalent? Could someone please help me understand these differences?**
  
  **Tags:** neural-networks, backpropagation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-22. Score: 2
  
  The key idea of [backpropagation](https://en.wikipedia.org/wiki/Backpropagation) can be summarized as:
  
  > Each individual component of the gradient, $\partial C/\partial w\_{jk}^{l}$ can be computed by the chain rule; but doing this separately for each weight is inefficient. Backpropagation efficiently computes the gradient by avoiding duplicate calculations and not computing unnecessary intermediate values, by computing the gradient of each layer – specifically the gradient of the weighted input of each layer, denoted by $\delta^{l}$ – from back to front... The $\delta ^{l}$ can easily be computed recursively, going from right to left, as: $$\delta ^{l-1}:=(f^{l-1})'\circ (W^{l})^{T}\cdot \delta ^{l}$$
  > The gradients of the weights can thus be computed using a few matrix multiplications for each level; this is backpropagation.
  
  Once this crucial step is done, all the weights can be straightforwardly updated via the delta rule involving a learning rate. Therefore you 1st article's quoted section/code just essentially implements the above crucial gradients $\delta ^{l-1}$ of the weighted inputs of each layer from code line 99-111, and code line 113-125 implements the final weights delta update rule which additionally involves the multiplicative terms of learning rate and the respective weighted inputs of each layer in Numpy's matrix/array linear algebra way.
  
  And the error signal of your 2nd article's quoted section ("the error signal for neurons in the hidden layer is accumulated from neurons in the output layer") is just the inner function part $(W^{l})^{T}\cdot \delta ^{l}$ of the composed gradient function $\delta ^{l-1}$. And at the end line 18 of the quoted code snippet you have:
  
  `neuron['delta'] = errors[j] * transfer_derivative(neuron['output'])`
  
  Thus the same gradient of each layer $\delta ^{l-1}$ as expressed as `neuron['delta']` here is obtained too after multiplied by the derivative of the activation/transfer function though here seems only primitive Python data types are employed instead of Numpy's linear algebra data types.
  
  > **Comments:**
  > **username_0**: Thanks for explaining!  With your explanation, both articles seem similar.  So this means that to calculate the delta of a neuron in the hidden layers, in both ways, both the first article (which uses the current layer) and the second article (which uses the next layer), work equivalently and are based on the same principles for the same goal?, even using a slightly different way? Did I understand correctly?  
  > **username_1**: Yes, your conclusion is correct, there’s only one standard back propagation algorithm though there’re many different implementations such as manifested here and in another similar question you asked recently. One idea with many different implementations is a common phenomenon in applied sciences and engineering.  
  > **username_0**: Thanks!. But, in second article, the sum of DELTA OUTPUT * WEIGHT to calcule delta in hidden layer, he explains that WEIGHT is the weight J (the index of the neuron J of current hidden layer in vector of the output layer, referring to input J), that is, the connection weight of output layer, that is multiplied by input of output layer (the input of the output layer is the output of the last hidden layer). But, in first article is different, because the weights are the weights that connect the input layer input with the current hidden layer. I had this question below:  
  > **username_0**: I have this another question about the first article: the first article multiplies the delta of the output layer with the weight matrix of the self hidden layer(that is, the weight matrix of the current hidden layer L itself). In contrast, the second article, when calculating the delta in the hidden layer, using a sum, it multiplies the delta of the output layer itself with the weights of the neurons in the output layer itself(Formula sum of DELTA OUPUT * WEIGHT OUPUT). It gets little confusing, and seems to be different calculation. Please, could help me clarify this point?  
  > **username_1**: The 1st article is using matrix algebra just as my above answer for the hidden layer's all units' gradients $\delta^{l-1}$ of their respective weighted input, so if 3 units in the current hidden layer $\delta^{l-1}$ is a vector of 3 gradients. This [ref](https://home.agh.edu.pl/%7Evlsi/AI/backp_t_en/backprop.html) is clearer, the first hidden layer has 3 units (1,2,3), each has gradient of its own weighted input, and it's not in matrix form, so like your 2nd article, you can compare and check its for-loops carefully though this is mainly a coding practice, I'd prefer 1st article's matrix code.  
  > **username_0**: Thank for the explanations!. It has helped me a lot to understand the concepts.  
  
  ---
  
  