Answers.Body: <p>This is just an application of the chain rule. The same chapter has "Proof of the four fundamental equations" section, which proves BP1-2, while PB3-4 are left as exercise to the reader. I agree that it's a good exercise indeed, that's why I encourage you to <strong>stop here and try to prove it yourself using a chain rule</strong>.</p>
  
  <p>Now, if you decided to read further, here's the sketch of a proof. </p>
  
  <p>Recall equations (25) and (29), both <em>definitions</em>:</p>
  
  <ul>
  <li><p><span class="math-container">$z^l$</span> is a linear transformation of <span class="math-container">$a^{l-1}$</span>: <span class="math-container">$z^l = w^l a^{l-1} + b^l$</span></p></li>
  <li><p><span class="math-container">$\delta^l$</span> is a partial derivative of <span class="math-container">$C$</span> with respect to <span class="math-container">$z^l$</span>: <span class="math-container">$\delta^l  = \dfrac{\partial C}{\partial z^l}$</span></p></li>
  </ul>
  
  <p>The chain rule itself:</p>
  
  <ul>
  <li>the partial derivative of <span class="math-container">$C$</span> with respect to <span class="math-container">$b^l$</span>: </li>
  </ul>
  
  <p><span class="math-container">$$\frac{\partial C}{\partial b^l_j} = \sum_k \frac{\partial C}{\partial z^l_k} \frac{\partial z^l_k}{\partial b^l_j}$$</span></p>
  
  <p>Almost all elements in this sum will be zero, except for one when <span class="math-container">$k=j$</span>.
  The first term in it is by definition <span class="math-container">$\delta^l_j$</span>, the second term <span class="math-container">$\dfrac{\partial z^l_j}{\partial b^l_j}$</span> is <span class="math-container">$1$</span>, because <span class="math-container">$z$</span> is linear with respect to <span class="math-container">$b$</span>.</p>
  
  <p>The derivative with respect to weights is taken the same way, and differs only in the last term: <span class="math-container">$z$</span> is linear with respect to <span class="math-container">$w$</span> as well, but with a coefficient <span class="math-container">$a^{l-1}$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-10-01T16:05:03.937Z
Answers.LastActivityDate: 2018-11-26T18:25:54.667Z
Answers.LastEditDate: 2018-11-26T18:25:54.667Z
Answers.UserDisplayName: Maxim
Body: <p>While reading <a href="http://neuralnetworksanddeeplearning.com/chap2.html" rel="nofollow noreferrer">the book on neural networks</a> by Michael Nielson, I had a problem understanding equation (BP3), which is</p>
  <p><span class="math-container">$$
  \frac{\partial C}{\partial b_{j}^{l}}=\delta_{j}^{l} \tag{BP3}\label{BP3},
  $$</span>
  which can be translated to plain English as follows</p>
  <blockquote>
  <p>The change in cost, <span class="math-container">$C$</span>, with respect to the bias of a neuron, <span class="math-container">$b_{j}^{l}$</span>, is equal to the error in that neuron, <span class="math-container">$\delta_{j}^{l}$</span>.</p>
  </blockquote>
  <p>Why would this be the case?</p>
  
Tags: <neural-networks><deep-learning><backpropagation><math><calculus>
Title: Why is the change in cost wrt bias in neural network equal to error in the neuron?
UserDisplayName: Madhusoodan P
ThreadText: # Why is the change in cost wrt bias in neural network equal to error in the neuron?
  
  (Asked by: username_0 on 2017-04-21)
  
  While reading [the book on neural networks](http://neuralnetworksanddeeplearning.com/chap2.html) by Michael Nielson, I had a problem understanding equation (BP3), which is
  
  $$
  \frac{\partial C}{\partial b\_{j}^{l}}=\delta\_{j}^{l} \tag{BP3}\label{BP3},
  $$
  which can be translated to plain English as follows
  
  > The change in cost, $C$, with respect to the bias of a neuron, $b\_{j}^{l}$, is equal to the error in that neuron, $\delta\_{j}^{l}$.
  
  Why would this be the case?
  
  **Tags:** neural-networks, deep-learning, backpropagation, math, calculus
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2017-10-01. Score: 1
  
  This is just an application of the chain rule. The same chapter has "Proof of the four fundamental equations" section, which proves BP1-2, while PB3-4 are left as exercise to the reader. I agree that it's a good exercise indeed, that's why I encourage you to **stop here and try to prove it yourself using a chain rule**.
  
  Now, if you decided to read further, here's the sketch of a proof.
  
  Recall equations (25) and (29), both *definitions*:
  
  * $z^l$ is a linear transformation of $a^{l-1}$: $z^l = w^l a^{l-1} + b^l$
  * $\delta^l$ is a partial derivative of $C$ with respect to $z^l$: $\delta^l = \dfrac{\partial C}{\partial z^l}$
  
  The chain rule itself:
  
  * the partial derivative of $C$ with respect to $b^l$:
  
  $$\frac{\partial C}{\partial b^l\_j} = \sum\_k \frac{\partial C}{\partial z^l\_k} \frac{\partial z^l\_k}{\partial b^l\_j}$$
  
  Almost all elements in this sum will be zero, except for one when $k=j$.
  The first term in it is by definition $\delta^l\_j$, the second term $\dfrac{\partial z^l\_j}{\partial b^l\_j}$ is $1$, because $z$ is linear with respect to $b$.
  
  The derivative with respect to weights is taken the same way, and differs only in the last term: $z$ is linear with respect to $w$ as well, but with a coefficient $a^{l-1}$.
  
  
  ---
  
  