Answers.Body: <p>I really like the way Michael Nielsen describes backpropagation in his <a href="http://neuralnetworksanddeeplearning.com/index.html" rel="nofollow noreferrer">book</a>. I will not explain all the details of these formulas because for that it's way better to read his <a href="http://neuralnetworksanddeeplearning.com/chap2.html" rel="nofollow noreferrer">chapter on backpropagation</a>.</p>
  <p>The formulas are the following:</p>
  <p><span class="math-container">\begin{align*}
  (1) \quad &amp; \delta^L = \Delta_a C \odot \sigma'(z^L) \\
  (2) \quad &amp; \delta^l = ((w_k^{l+1})^T\delta^{l+1}) \odot \sigma'(z^l)\\
  (3) \quad &amp; \frac{\partial C}{\partial b^l_j}=  \delta^l_j \\
  (4) \quad &amp; \frac{\partial C}{\partial w^l_{j \space k}}=  a_k^{l-1} \delta^l_j \\
  \end{align*}</span></p>
  <p>Based on these formulas, the derivative of the cost function with respect to the bias is the derivative of the current layer's activation function with respect to the cost function, which in these formulas is represented by the <span class="math-container">$\delta^l$</span> variable. <strong>So to answer your question, no it doesn't use the sum of all Deltas of the layer</strong>. That would be a constraint we wouldn't want for the network because it would mean the derivative of all the biases in the network would have the same change factor. We want all the variables to be able to change independently of each other.Thats why we only use the neuron's Delta.</p>
  <p>When we consider the sum of gradients, it is when performing gradient descent with multiple data points. Because the cost function is typically a sum of all the errors, we are supposed to calculate the gradient of that summation.</p>
  <p>Let's say we are using the mean squared error cost function:</p>
  <p><span class="math-container">$$
  C =  \frac{1}{M}\sum_{i = 1}^{M} (y'_i - y_i)^2
  $$</span></p>
  <p>When differentiating with respect to the output of the last layer we get:</p>
  <p><span class="math-container">$$
  \frac{\partial C}{\partial y'} = \frac{\partial}{\partial y_i'} \frac{1}{M} (\sum_{i = 1}^{M} (y'_i - y_i)^2)
  $$</span>
  But because we can exchange the summation of derivatives with the derivatives of a summation (because derivation is a linear transformation), we can calculate the gradients of each weight first and then do the summation:
  <span class="math-container">$$
  \frac{\partial C}{\partial y'} =   \frac{1}{M}\sum_{i = 1}^{M} \frac{\partial}{\partial y_i'} (y'_i - y_i)^2
  $$</span></p>
  <p>If you want another resource to see how backpropagation can be implemented, you can read the following <a href="https://github.com/mnielsen/neural-networks-and-deep-learning/blob/master/src/network.py" rel="nofollow noreferrer">code</a> from the same author I mentioned before. Here they implement what I just said - they first calculate the gradients with respect to all the weights and then for the SGD update calculate the sum of the gradients over all the data points.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-03T20:58:59.350Z
Answers.Text: In this link, The author Makes the sum of each deltas of each neuron in each layer, during the mini-bath. So, this results in several "accumulated deltas" of each neuron in each layer. Did I understand the article correctly? So that each delta of each individual neuron will be accumulated during the mini-bath? And only then will he do the SGD? Am I correct?
Answers.UserDisplayName: will The J
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-03T22:50:02.933Z
Answers.Text: Correct. We first get the accumulation of the deltas for each neuron in the mini-batch and then use that to update the weights. A common error I had when learning backpropagation was thinking that for each forward pass we do a backward pass, but it's not for each forward pass but for each batch. This is like this because the cost function must try to minimize all the errors of the data points, not only a single point. That's why we get the summation. Remember that it is not the same to update the weights with the sum of the gradients as updating them for each gradient independently.
Answers.UserDisplayName: Cesar Ruiz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-03T23:02:08.313Z
Answers.Text: This means that we never use the gradients calculated for each data point alone. Technically, to update the weights with gradient descent we must take the sum of the gradients across all data points. However, because it can be computationally expensive to do that, we perform gradient descent stochastically on small groups (batches) of the data using SGD.
Answers.UserDisplayName: Cesar Ruiz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-04T00:26:14.040Z
Answers.Text: Thank you for clarifying these points for me. This helps me
Answers.UserDisplayName: will The J
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-03T19:55:46.370Z
Answers.LastActivityDate: 2024-01-03T19:55:46.370Z
Answers.UserDisplayName: Cesar Ruiz
Body: <p>In multilayer perceptron, to update the Bias value, do we use the neuron's delta, or do we use the sum of all deltas?</p>
  <p>I ask this because I once saw a video on YouTube where the formula for updating the neuron's Bias in backpropagation, The correction factor to update Bias was a sum of all the deltas (of the layer).</p>
  <p>But, in other videos and articles, such as this article here: <strong><a href="https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/" rel="nofollow noreferrer">https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/</a></strong>, the bias correction factor for the neuron is the delta itself. This is because Bias has no input, so its correction factor was just the neuron's delta.</p>
  <p><strong>I was in doubt: In multilayer perceptron, to update the Bias value, do we use the neuron's delta, or do we use the sum of all deltas?</strong></p>
  
Tags: <neural-networks><backpropagation><bias>
Title: In multilayer perceptron, to update the Bias value, do we use the neuron's delta, or do we use the sum of all deltas?
UserDisplayName: will The J
ThreadText: # In multilayer perceptron, to update the Bias value, do we use the neuron's delta, or do we use the sum of all deltas?
  
  (Asked by: username_0 on 2024-01-02)
  
  In multilayer perceptron, to update the Bias value, do we use the neuron's delta, or do we use the sum of all deltas?
  
  I ask this because I once saw a video on YouTube where the formula for updating the neuron's Bias in backpropagation, The correction factor to update Bias was a sum of all the deltas (of the layer).
  
  But, in other videos and articles, such as this article here: **<https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/>**, the bias correction factor for the neuron is the delta itself. This is because Bias has no input, so its correction factor was just the neuron's delta.
  
  **I was in doubt: In multilayer perceptron, to update the Bias value, do we use the neuron's delta, or do we use the sum of all deltas?**
  
  **Tags:** neural-networks, backpropagation, bias
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-03. Score: 1 | Accepted answer
  
  I really like the way Michael Nielsen describes backpropagation in his [book](http://neuralnetworksanddeeplearning.com/index.html). I username_0 not explain all the details of these formulas because for that it's way better to read his [chapter on backpropagation](http://neuralnetworksanddeeplearning.com/chap2.html).
  
  The formulas are the following:
  
  \begin{align\*}
  (1) \quad & \delta^L = \Delta\_a C \odot \sigma'(z^L) \\
  (2) \quad & \delta^l = ((w\_k^{l+1})^T\delta^{l+1}) \odot \sigma'(z^l)\\
  (3) \quad & \frac{\partial C}{\partial b^l\_j}= \delta^l\_j \\
  (4) \quad & \frac{\partial C}{\partial w^l\_{j \space k}}= a\_k^{l-1} \delta^l\_j \\
  \end{align\*}
  
  Based on these formulas, the derivative of the cost function with respect to the bias is the derivative of the current layer's activation function with respect to the cost function, which in these formulas is represented by the $\delta^l$ variable. **So to answer your question, no it doesn't use the sum of all Deltas of the layer**. That would be a constraint we wouldn't want for the network because it would mean the derivative of all the biases in the network would have the same change factor. We want all the variables to be able to change independently of each other.Thats why we only use the neuron's Delta.
  
  When we consider the sum of gradients, it is when performing gradient descent with multiple data points. Because the cost function is typically a sum of all the errors, we are supposed to calculate the gradient of that summation.
  
  Let's say we are using the mean squared error cost function:
  
  $$
  C = \frac{1}{M}\sum\_{i = 1}^{M} (y'\_i - y\_i)^2
  $$
  
  When differentiating with respect to the output of the last layer we get:
  
  $$
  \frac{\partial C}{\partial y'} = \frac{\partial}{\partial y\_i'} \frac{1}{M} (\sum\_{i = 1}^{M} (y'\_i - y\_i)^2)
  $$
  But because we can exchange the summation of derivatives with the derivatives of a summation (because derivation is a linear transformation), we can calculate the gradients of each weight first and then do the summation:
  $$
  \frac{\partial C}{\partial y'} = \frac{1}{M}\sum\_{i = 1}^{M} \frac{\partial}{\partial y\_i'} (y'\_i - y\_i)^2
  $$
  
  If you want another resource to see how backpropagation can be implemented, you can read the following [code](https://github.com/mnielsen/neural-networks-and-deep-learning/blob/master/src/network.py) from the same author I mentioned before. Here they implement what I just said - they first calculate the gradients with respect to all the weights and then for the SGD update calculate the sum of the gradients over all the data points.
  
  > **Comments:**
  > **username_0**: In this link, The author Makes the sum of each deltas of each neuron in each layer, during the mini-bath. So, this results in several "accumulated deltas" of each neuron in each layer. Did I understand the article correctly? So that each delta of each individual neuron username_0 be accumulated during the mini-bath? And only then username_0 he do the SGD? Am I correct?  
  > **username_1**: Correct. We first get the accumulation of the deltas for each neuron in the mini-batch and then use that to update the weights. A common error I had when learning backpropagation was thinking that for each forward pass we do a backward pass, but it's not for each forward pass but for each batch. This is like this because the cost function must try to minimize all the errors of the data points, not only a single point. That's why we get the summation. Remember that it is not the same to update the weights with the sum of the gradients as updating them for each gradient independently.  
  > **username_1**: This means that we never use the gradients calculated for each data point alone. Technically, to update the weights with gradient descent we must take the sum of the gradients across all data points. However, because it can be computationally expensive to do that, we perform gradient descent stochastically on small groups (batches) of the data using SGD.  
  > **username_0**: Thank you for clarifying these points for me. This helps me  
  
  ---
  
  