Answers.Body: <p>I believe he's just saying that:</p>
  
  <p><span class="math-container">$$
  \frac{\partial C}{\partial z_j^l} \Delta z_j^l \approx \frac{\partial C}{\partial z_j^l} \partial z_j^l \approx \partial C
  $$</span></p>
  
  <p>so that the change in cost function can be arrived at simply for a small enough perturbation <span class="math-container">$\Delta z_j^l$</span>.</p>
  
  <p>Or, taking that line of approximations backwards, the change in the cost function for a given perturbation is just:
  <span class="math-container">$$
  \partial C \approx \frac{\partial C}{\partial z_j^l} \partial z_j^l \approx  \frac{\partial C}{\partial z_j^l} \Delta z_j^l
  $$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-05T15:04:43.637Z
Answers.Text: The $\approx$ in $\frac{\partial C}{\partial z_j^l} \partial z_j^l \approx \partial C$ can be replaced with an $=$ (it is not just an approximation, it is an exact equation).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-05T15:09:30.367Z
Answers.Text: How does this answer the question? You're just approximating $\partial C$. It doesn't not explain why, if you add $\Delta z$ to $z$, then the derivative of $C$ w.r.t. $z$ needs to be multiplied by $\Delta z$.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-05T15:20:28.437Z
Answers.Text: @nbro To respond to your first comment: Well, I wrote it like that because [it's not a very mathematically rigorous thing to do to write it with strict equality](https://www.khanacademy.org/math/ap-calculus-ab/ab-differential-equations-new/ab-7-6/v/addressing-treating-differentials-algebraically). It's probably OK here, but I haven't dug into it.
Answers.UserDisplayName: Peter K.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-05T15:21:53.887Z
Answers.Text: @nbro I'm confused by your second comment: `It doesn't not explain why` ??? Not sure what you're saying.
Answers.UserDisplayName: Peter K.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-05T15:22:02.107Z
Answers.Text: Why isn't it a mathematically rigorous thing? It is just another way of writing $\frac{\partial C}{\partial z} = \frac{\partial C}{\partial z}$.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-05T15:24:58.443Z
Answers.Text: @nbro See [this question and its answers over on SE.ME](https://math.stackexchange.com/q/47092/5965).
Answers.UserDisplayName: Peter K.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-05T15:25:12.810Z
Answers.Text: If you look at the context of the question, you will that the deamon adds $\Delta z$ to $z$, so Nielsen (the author of that article) states that "the overall cost changes by $\frac{\Delta C}{\partial z}\Delta z$". First of all, it is not clear what he means by "overall cost". Second, I think that your answer does not explain the relation between the addition of $\Delta z$ to $z$ and the fact that the "overall cost" is now $\frac{\Delta C}{\partial z}\Delta z$.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-05T15:29:22.570Z
Answers.Text: Ok, thanks for pointing me to that question. I was not aware of this. Anyway, a lot of people do it. For example, in integration by substitution, one manipulates infenitesimals.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-04T16:01:08.137Z
Answers.LastActivityDate: 2019-06-04T16:01:08.137Z
Answers.UserDisplayName: Peter K.
Answers.Body: <p>I think that Nielsen just wanted to convey the idea of the back-propagation algorithm using that formula, as you can read from the next paragraph "Now, this demon is a good demon...", so I don't think that that partial derivative is mathematically correct, provided the partial derivative is still with respect to <span class="math-container">$z_j^l$</span>.</p>
  
  <p><span class="math-container">$C$</span> is the cost (or loss) function. <span class="math-container">$z_j^l$</span> is the linear output of neuron <span class="math-container">$j$</span> in layer <span class="math-container">$l$</span>, which is followed by a non-linear function (e.g. sigmoid), denoted by <span class="math-container">$\sigma$</span>. So, the actual output of neuron <span class="math-container">$j$</span> in layer <span class="math-container">$l$</span> is <span class="math-container">$\sigma(z_j^l)$</span>. </p>
  
  <p>The partial derivative of the cost function <span class="math-container">$C$</span> with respect to this neuron's linear output, <span class="math-container">$z_j^l$</span>, is <span class="math-container">$$\frac{\partial C}{\partial z_j^l} = \frac{\partial C}{\partial z_j^l} 1 = \frac{\partial C}{\partial z_j^l} \frac{\partial z_j^l}{\partial z_j^l}.$$</span></p>
  
  <p>If the the linear output of node <span class="math-container">$j$</span> in layer <span class="math-container">$l$</span> is now <span class="math-container">$z_j^l + \Delta z_j^l$</span>, then the partial derivative with respect to <span class="math-container">$z_j^l$</span> becomes</p>
  
  <p><span class="math-container">\begin{align}
  \frac{\partial C}{\partial z_j^l} 
  &amp;= \frac{\partial C}{\partial (z_j^l + \Delta z_j^l)}\frac{\partial (z_j^l + \Delta z_j^l)}{\partial z_j^l}  \\
  &amp;= \frac{\partial C}{\partial (z_j^l + \Delta z_j^l)} \left( \frac{\partial z_j^l}{\partial z_j^l} +  \frac{\partial \Delta z_j^l}{\partial z_j^l} \right) \\
  &amp;= \frac{\partial C}{\partial (z_j^l + \Delta z_j^l)} \left( 1 +  \frac{\partial \Delta z_j^l}{\partial z_j^l} \right) \\
  &amp;= \frac{\partial C}{\partial (z_j^l + \Delta z_j^l)} +  \frac{\partial C}{\partial (z_j^l + \Delta z_j^l)}\frac{\partial \Delta z_j^l}{\partial z_j^l} \\
  \end{align}</span></p>
  
  <p><span class="math-container">$\Delta z_j^l$</span> depends on <span class="math-container">$z_j^l$</span>, but it is not specified how.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-04T15:39:18.310Z
Answers.LastActivityDate: 2019-06-04T15:39:18.310Z
Answers.UserDisplayName: nbro
Body: <p>I was reading the following book: <a href="http://neuralnetworksanddeeplearning.com/chap2.html" rel="nofollow noreferrer">http://neuralnetworksanddeeplearning.com/chap2.html</a></p>
  
  <p>and towards the end of equation 29, there is a paragraph that explains this: </p>
  
  <p><a href="https://i.sstatic.net/KkBdn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KkBdn.png" alt="enter image description here"></a></p>
  
  <p>However I am unsure how the equation below is derived:</p>
  
  <p><a href="https://i.sstatic.net/DlOfT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/DlOfT.png" alt="enter image description here"></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-05T14:52:24.160Z
Comments.Text: The expression you give in your final statement is the definition of a math derivative. Partial, because cost depends on all other network parameters, and using $\Delta$ instead of differential, but it is the concept of derivative:change in result due to infinitesimal change in input.
Comments.UserDisplayName: pasaba por aqui
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-05T15:01:07.873Z
Comments.Text: @pasabaporaqui It's not the definition of derivative. The problem with that notation is that if you change a parameter by a certain additive amount the derivative will not change by the same multiplicative factor (in general).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-05T15:05:57.770Z
Comments.Text: I think for state functions where you can write down the total differential, using Euler's Rule and the Chain Rule, C depends upon z. So, the infinitesimal change in C, dC in some arbitrary dimension is given by:
  
  dC = (delta C/delta z)*dz
Comments.UserDisplayName: THAT_AI_GUY
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-05T15:07:13.090Z
Comments.Text: @nbro: $ df = f' dx$, $ \Delta f = f' \Delta x$, $f' = lim ( \Delta x -> 0 ) \Delta f / \Delta x$, ... lots of ways to write same concept: change in result due to small change in input.
Comments.UserDisplayName: pasaba por aqui
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-05T15:12:02.683Z
Comments.Text: @pasabaporaqui Ok. However, you have to look at the context. Nielsen states that if you add $\Delta z$ to $z$, then the derivative changes by the same multiplicative factor, but this is not, in general, true. Furthemore, the text is really ambigous. He states "the overall cost". The overall cost of what?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-05T15:20:49.127Z
Comments.Text: I think it works for this case because C is only dependent on z. So if you have a graph of cost vs. z, then a small change in z results in a small change in C. Thus, the total (overall) derivative of C wrt z is equivalent to the partial derivative of C wrt z. So dC/dz = delta C/delta z. Hence, dC = (delta C/delta z)*dz if you multiply both sides by dz.
Comments.UserDisplayName: THAT_AI_GUY
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-05T15:30:05.700Z
Comments.Text: C does not only depend on z. The network has a lot of nodes in that example.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-05T15:42:48.870Z
Comments.Text: C in this example is a function of a single variable (z)
Comments.UserDisplayName: THAT_AI_GUY
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-05T16:04:57.897Z
Comments.Text: @flying_costa No, it is not. Why do you think so? $C$ is the cost function and it is a function of all variables of the neural network.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-05T16:38:52.257Z
Comments.Text: @flying_costa: there are several usual ways to generalize same concept to functions of several arguments: partial derivative, gradient ( $\nabla $) ... In the text you quote, partial derivative is used for all parameters $z_i^j$ with $i$ in ... and $j$ in ....
Comments.UserDisplayName: pasaba por aqui
Tags: <neural-networks><backpropagation>
Title: How does adding a small change to an neuron's weighted input affect the overall cost?
UserDisplayName: THAT_AI_GUY
ThreadText: # How does adding a small change to an neuron's weighted input affect the overall cost?
  
  (Asked by: username_0 on 2019-06-04)
  
  I was reading the following book: <http://neuralnetworksanddeeplearning.com/chap2.html>
  
  and towards the end of equation 29, there is a paragraph that explains this:
  
  [![enter image description here](https://i.sstatic.net/KkBdn.png)](https://i.sstatic.net/KkBdn.png)
  
  However I am unsure how the equation below is derived:
  
  [![enter image description here](https://i.sstatic.net/DlOfT.png)](https://i.sstatic.net/DlOfT.png)
  
  **Tags:** neural-networks, backpropagation
  
  > **Comments:**
  > **username_1**: The expression you give in your final statement is the definition of a math derivative. Partial, because cost depends on all other network parameters, and using $\Delta$ instead of differential, but it is the concept of derivative:change in result due to infinitesimal change in input.  
  > **username_2**: @username_1 It's not the definition of derivative. The problem with that notation is that if you change a parameter by a certain additive amount the derivative will not change by the same multiplicative factor (in general).  
  > **username_0**: I think for state functions where you can write down the total differential, using Euler's Rule and the Chain Rule, C depends upon z. So, the infinitesimal change in C, dC in some arbitrary dimension is given by:  dC = (delta C/delta z)*dz  
  > **username_1**: @username_2: $ df = f' dx$, $ \Delta f = f' \Delta x$, $f' = lim ( \Delta x -> 0 ) \Delta f / \Delta x$, ... lots of ways to write same concept: change in result due to small change in input.  
  > **username_2**: @username_1 Ok. However, you have to look at the context. Nielsen states that if you add $\Delta z$ to $z$, then the derivative changes by the same multiplicative factor, but this is not, in general, true. Furthemore, the text is really ambigous. He states "the overall cost". The overall cost of what?  
  > **username_0**: I think it works for this case because C is only dependent on z. So if you have a graph of cost vs. z, then a small change in z results in a small change in C. Thus, the total (overall) derivative of C wrt z is equivalent to the partial derivative of C wrt z. So dC/dz = delta C/delta z. Hence, dC = (delta C/delta z)*dz if you multiply both sides by dz.  
  > **username_2**: C does not only depend on z. The network has a lot of nodes in that example.  
  > **username_0**: C in this example is a function of a single variable (z)  
  > **username_2**: @flying_costa No, it is not. Why do you think so? $C$ is the cost function and it is a function of all variables of the neural network.  
  > **username_1**: @flying_costa: there are several usual ways to generalize same concept to functions of several arguments: partial derivative, gradient ( $\nabla $) ... In the text you quote, partial derivative is used for all parameters $z_i^j$ with $i$ in ... and $j$ in ....  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2019-06-04. Score: 1
  
  I believe he's just saying that:
  
  $$
  \frac{\partial C}{\partial z\_j^l} \Delta z\_j^l \approx \frac{\partial C}{\partial z\_j^l} \partial z\_j^l \approx \partial C
  $$
  
  so that the change in cost function can be arrived at simply for a small enough perturbation $\Delta z\_j^l$.
  
  Or, taking that line of approximations backwards, the change in the cost function for a given perturbation is just:
  $$
  \partial C \approx \frac{\partial C}{\partial z\_j^l} \partial z\_j^l \approx \frac{\partial C}{\partial z\_j^l} \Delta z\_j^l
  $$
  
  > **Comments:**
  > **username_2**: The $\approx$ in $\frac{\partial C}{\partial z_j^l} \partial z_j^l \approx \partial C$ can be replaced with an $=$ (it is not just an approximation, it is an exact equation).  
  > **username_2**: How does this answer the question? You're just approximating $\partial C$. It doesn't not explain why, if you add $\Delta z$ to $z$, then the derivative of $C$ w.r.t. $z$ needs to be multiplied by $\Delta z$.  
  > **username_3**: @username_2 To respond to your first comment: Well, I wrote it like that because [it's not a very mathematically rigorous thing to do to write it with strict equality](https://www.khanacademy.org/math/ap-calculus-ab/ab-differential-equations-new/ab-7-6/v/addressing-treating-differentials-algebraically). It's probably OK here, but I haven't dug into it.  
  > **username_3**: @username_2 I'm confused by your second comment: `It doesn't not explain why` ??? Not sure what you're saying.  
  > **username_2**: Why isn't it a mathematically rigorous thing? It is just another way of writing $\frac{\partial C}{\partial z} = \frac{\partial C}{\partial z}$.  
  > **username_3**: @username_2 See [this question and its answers over on SE.ME](https://math.stackexchange.com/q/47092/5965).  
  > **username_2**: If you look at the context of the question, you will that the deamon adds $\Delta z$ to $z$, so Nielsen (the author of that article) states that "the overall cost changes by $\frac{\Delta C}{\partial z}\Delta z$". First of all, it is not clear what he means by "overall cost". Second, I think that your answer does not explain the relation between the addition of $\Delta z$ to $z$ and the fact that the "overall cost" is now $\frac{\Delta C}{\partial z}\Delta z$.  
  > **username_2**: Ok, thanks for pointing me to that question. I was not aware of this. Anyway, a lot of people do it. For example, in integration by substitution, one manipulates infenitesimals.  
  
  ---
  
  ### Answer by username_2 on 2019-06-04. Score: 0
  
  I think that Nielsen just wanted to convey the idea of the back-propagation algorithm using that formula, as you can read from the next paragraph "Now, this demon is a good demon...", so I don't think that that partial derivative is mathematically correct, provided the partial derivative is still with respect to $z\_j^l$.
  
  $C$ is the cost (or loss) function. $z\_j^l$ is the linear output of neuron $j$ in layer $l$, which is followed by a non-linear function (e.g. sigmoid), denoted by $\sigma$. So, the actual output of neuron $j$ in layer $l$ is $\sigma(z\_j^l)$.
  
  The partial derivative of the cost function $C$ with respect to this neuron's linear output, $z\_j^l$, is $$\frac{\partial C}{\partial z\_j^l} = \frac{\partial C}{\partial z\_j^l} 1 = \frac{\partial C}{\partial z\_j^l} \frac{\partial z\_j^l}{\partial z\_j^l}.$$
  
  If the the linear output of node $j$ in layer $l$ is now $z\_j^l + \Delta z\_j^l$, then the partial derivative with respect to $z\_j^l$ becomes
  
  \begin{align}
  \frac{\partial C}{\partial z\_j^l}
  &= \frac{\partial C}{\partial (z\_j^l + \Delta z\_j^l)}\frac{\partial (z\_j^l + \Delta z\_j^l)}{\partial z\_j^l} \\
  &= \frac{\partial C}{\partial (z\_j^l + \Delta z\_j^l)} \left( \frac{\partial z\_j^l}{\partial z\_j^l} + \frac{\partial \Delta z\_j^l}{\partial z\_j^l} \right) \\
  &= \frac{\partial C}{\partial (z\_j^l + \Delta z\_j^l)} \left( 1 + \frac{\partial \Delta z\_j^l}{\partial z\_j^l} \right) \\
  &= \frac{\partial C}{\partial (z\_j^l + \Delta z\_j^l)} + \frac{\partial C}{\partial (z\_j^l + \Delta z\_j^l)}\frac{\partial \Delta z\_j^l}{\partial z\_j^l} \\
  \end{align}
  
  $\Delta z\_j^l$ depends on $z\_j^l$, but it is not specified how.
  
  
  ---
  
  