<p>This is just an application of the chain rule. The same chapter has "Proof of the four fundamental equations" section, which proves BP1-2, while PB3-4 are left as exercise to the reader. I agree that it's a good exercise indeed, that's why I encourage you to <strong>stop here and try to prove it yourself using a chain rule</strong>.</p>

<p>Now, if you decided to read further, here's the sketch of a proof. </p>

<p>Recall equations (25) and (29), both <em>definitions</em>:</p>

<ul>
<li><p><span class="math-container">$z^l$</span> is a linear transformation of <span class="math-container">$a^{l-1}$</span>: <span class="math-container">$z^l = w^l a^{l-1} + b^l$</span></p></li>
<li><p><span class="math-container">$\delta^l$</span> is a partial derivative of <span class="math-container">$C$</span> with respect to <span class="math-container">$z^l$</span>: <span class="math-container">$\delta^l  = \dfrac{\partial C}{\partial z^l}$</span></p></li>
</ul>

<p>The chain rule itself:</p>

<ul>
<li>the partial derivative of <span class="math-container">$C$</span> with respect to <span class="math-container">$b^l$</span>: </li>
</ul>

<p><span class="math-container">$$\frac{\partial C}{\partial b^l_j} = \sum_k \frac{\partial C}{\partial z^l_k} \frac{\partial z^l_k}{\partial b^l_j}$$</span></p>

<p>Almost all elements in this sum will be zero, except for one when <span class="math-container">$k=j$</span>.
The first term in it is by definition <span class="math-container">$\delta^l_j$</span>, the second term <span class="math-container">$\dfrac{\partial z^l_j}{\partial b^l_j}$</span> is <span class="math-container">$1$</span>, because <span class="math-container">$z$</span> is linear with respect to <span class="math-container">$b$</span>.</p>

<p>The derivative with respect to weights is taken the same way, and differs only in the last term: <span class="math-container">$z$</span> is linear with respect to <span class="math-container">$w$</span> as well, but with a coefficient <span class="math-container">$a^{l-1}$</span>.</p>

CC BY-SA 4.0
2017-10-01T16:05:03.937Z
2018-11-26T18:25:54.667Z
2018-11-26T18:25:54.667Z
Maxim
<p>While reading <a href="http://neuralnetworksanddeeplearning.com/chap2.html" rel="nofollow noreferrer">the book on neural networks</a> by Michael Nielson, I had a problem understanding equation (BP3), which is</p>
<p><span class="math-container">$$
\frac{\partial C}{\partial b_{j}^{l}}=\delta_{j}^{l} \tag{BP3}\label{BP3},
$$</span>
which can be translated to plain English as follows</p>
<blockquote>
<p>The change in cost, <span class="math-container">$C$</span>, with respect to the bias of a neuron, <span class="math-container">$b_{j}^{l}$</span>, is equal to the error in that neuron, <span class="math-container">$\delta_{j}^{l}$</span>.</p>
</blockquote>
<p>Why would this be the case?</p>

<neural-networks><deep-learning><backpropagation><math><calculus>
Why is the change in cost wrt bias in neural network equal to error in the neuron?
Madhusoodan P
# Why is the change in cost wrt bias in neural network equal to error in the neuron?

(Asked by: username_0 on 2017-04-21)

While reading [the book on neural networks](http://neuralnetworksanddeeplearning.com/chap2.html) by Michael Nielson, I had a problem understanding equation (BP3), which is

$$
\frac{\partial C}{\partial b\_{j}^{l}}=\delta\_{j}^{l} \tag{BP3}\label{BP3},
$$
which can be translated to plain English as follows

> The change in cost, $C$, with respect to the bias of a neuron, $b\_{j}^{l}$, is equal to the error in that neuron, $\delta\_{j}^{l}$.

Why would this be the case?

**Tags:** neural-networks, deep-learning, backpropagation, math, calculus


---

## 1 answers

### Answer by username_1 on 2017-10-01. Score: 1

This is just an application of the chain rule. The same chapter has "Proof of the four fundamental equations" section, which proves BP1-2, while PB3-4 are left as exercise to the reader. I agree that it's a good exercise indeed, that's why I encourage you to **stop here and try to prove it yourself using a chain rule**.

Now, if you decided to read further, here's the sketch of a proof.

Recall equations (25) and (29), both *definitions*:

* $z^l$ is a linear transformation of $a^{l-1}$: $z^l = w^l a^{l-1} + b^l$
* $\delta^l$ is a partial derivative of $C$ with respect to $z^l$: $\delta^l = \dfrac{\partial C}{\partial z^l}$

The chain rule itself:

* the partial derivative of $C$ with respect to $b^l$:

$$\frac{\partial C}{\partial b^l\_j} = \sum\_k \frac{\partial C}{\partial z^l\_k} \frac{\partial z^l\_k}{\partial b^l\_j}$$

Almost all elements in this sum will be zero, except for one when $k=j$.
The first term in it is by definition $\delta^l\_j$, the second term $\dfrac{\partial z^l\_j}{\partial b^l\_j}$ is $1$, because $z$ is linear with respect to $b$.

The derivative with respect to weights is taken the same way, and differs only in the last term: $z$ is linear with respect to $w$ as well, but with a coefficient $a^{l-1}$.


---

