Answers.Body: <p>It's a matter of different loss functions used. In Andrew Ng's C1W2L09 video he derives the loss gradient as
  <span class="math-container">$$\frac{d}{dz} = \frac{d}{da}*\frac{da}{dz}$$</span>
  using the loss (cost) function <strong>binary cross entropy</strong> whose derivative is <span class="math-container">$\frac{d}{da} = -\frac{y}{a} + \frac{1 - y}{1 - a}$</span>
  and the activation function is sigmoid whose derivative is
  <span class="math-container">$\frac{da}{dz} = a(1-a)$</span>.
  The product of the two is
  <span class="math-container">$$a - y$$</span> which is consistent with the vectorized form you showed.</p>
  <p>Nielsen uses the <strong>quadratic loss function</strong> <span class="math-container">$L = \frac{1}{2}||y - a^L||^2$</span> whose derivative is as you've shown: <span class="math-container">$\frac{d}{da} = a^L - y$</span>. His formula <span class="math-container">$\delta^L = (a^L - y) \odot \sigma'(z^L)$</span> is specific to quadratic loss.</p>
  <p>So Andrew's <span class="math-container">$\frac{d}{dz} = \frac{d}{da}*\frac{da}{dz}$</span> is the same as Nielsen's matrix based <span class="math-container">$\delta^L=\nabla_aC \odot \sigma'(z^L)$</span>, here are the equivalent terms (Andrew's on the left, Nielsen's on the right)
  <span class="math-container">$$\frac{dL}{da} = \nabla_aC $$</span> and <span class="math-container">$$\frac{da}{dz} = \sigma'(z^L)$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-25T02:03:50.833Z
Answers.LastActivityDate: 2022-07-25T02:06:41.550Z
Answers.LastEditDate: 2022-07-25T02:06:41.550Z
Answers.UserDisplayName: rkuang25
Body: <p>I'm currently working through Week 5 of Andrew Ngs Machine Learning course on Coursera, which goes through the backprop algorithm for basic neural networks. Whilst trying to derive the formulae he gave in the lectures, I noticed that the formula for <span class="math-container">$\delta^L$</span>, &quot;error&quot; of last activation layer, is slightly different to that derived in <a href="http://neuralnetworksanddeeplearning.com/chap2.html" rel="nofollow noreferrer">http://neuralnetworksanddeeplearning.com/chap2.html</a>.</p>
  <p>In Andrew's, it seems like there is no inclusion of the partial derivative da/dz, or <span class="math-container">$\sigma'(z)$</span>, only the dC/da part.</p>
  <p><a href="https://i.sstatic.net/V7Gid.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/V7Gid.png" alt="Andrew Ng error term" /></a></p>
  <p>However Michael Nielson does include that term:</p>
  <p><a href="https://i.sstatic.net/g5tah.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/g5tah.png" alt="enter image description here" /></a></p>
  <p>Is this difference significant and why does it arise? Is it because the derivation Nielson goes through defines the Cost using the mean square errors, whereas Andrew Ng defines the cost using the -ylog(h(x))... one? Also will Nielson's equations score full marks on the Ng's assignment?</p>
  <p>Thank you for reading.</p>
  
Tags: <neural-networks><machine-learning><backpropagation>
Title: Discrepancy of backpropagation formula between Andrew Ngs ML Course and those derived by neuralnetworksanddeeplearning.com
UserDisplayName: James Liu
ThreadText: # Discrepancy of backpropagation formula between Andrew Ngs ML Course and those derived by neuralnetworksanddeeplearning.com
  
  (Asked by: username_0 on 2021-09-13)
  
  I'm currently working through Week 5 of Andrew Ngs Machine Learning course on Coursera, which goes through the backprop algorithm for basic neural networks. Whilst trying to derive the formulae he gave in the lectures, I noticed that the formula for $\delta^L$, "error" of last activation layer, is slightly different to that derived in <http://neuralnetworksanddeeplearning.com/chap2.html>.
  
  In Andrew's, it seems like there is no inclusion of the partial derivative da/dz, or $\sigma'(z)$, only the dC/da part.
  
  [![Andrew Ng error term](https://i.sstatic.net/V7Gid.png)](https://i.sstatic.net/V7Gid.png)
  
  However Michael Nielson does include that term:
  
  [![enter image description here](https://i.sstatic.net/g5tah.png)](https://i.sstatic.net/g5tah.png)
  
  Is this difference significant and why does it arise? Is it because the derivation Nielson goes through defines the Cost using the mean square errors, whereas Andrew Ng defines the cost using the -ylog(h(x))... one? Also will Nielson's equations score full marks on the Ng's assignment?
  
  Thank you for reading.
  
  **Tags:** neural-networks, machine-learning, backpropagation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-07-25. Score: 1
  
  It's a matter of different loss functions used. In Andrew Ng's C1W2L09 video he derives the loss gradient as
  $$\frac{d}{dz} = \frac{d}{da}\*\frac{da}{dz}$$
  using the loss (cost) function **binary cross entropy** whose derivative is $\frac{d}{da} = -\frac{y}{a} + \frac{1 - y}{1 - a}$
  and the activation function is sigmoid whose derivative is
  $\frac{da}{dz} = a(1-a)$.
  The product of the two is
  $$a - y$$ which is consistent with the vectorized form you showed.
  
  Nielsen uses the **quadratic loss function** $L = \frac{1}{2}||y - a^L||^2$ whose derivative is as you've shown: $\frac{d}{da} = a^L - y$. His formula $\delta^L = (a^L - y) \odot \sigma'(z^L)$ is specific to quadratic loss.
  
  So Andrew's $\frac{d}{dz} = \frac{d}{da}\*\frac{da}{dz}$ is the same as Nielsen's matrix based $\delta^L=\nabla\_aC \odot \sigma'(z^L)$, here are the equivalent terms (Andrew's on the left, Nielsen's on the right)
  $$\frac{dL}{da} = \nabla\_aC $$ and $$\frac{da}{dz} = \sigma'(z^L)$$
  
  
  ---
  
  