Answers.Body: <p><strong>TL;DR</strong>: This has to do with the way A. Ng has defined back propagation for the course. </p>
  
  <p><strong>Left Column</strong></p>
  
  <p>This is only with respect to <em>one</em> input example and so the <span class="math-container">$\frac{1}{m}$</span> factor reduces to 1 and can be omitted. He uses lower case to represent one input example (eg a vector <span class="math-container">$dz$</span>) and upper case with respect to a (mini-)batch (eg a matrix <span class="math-container">$dZ$</span>). </p>
  
  <p><strong>The <span class="math-container">$\frac{1}{m}$</span> factors in <span class="math-container">$dW,db$</span></strong></p>
  
  <p>In this definition of backprop, he "defers" multiplying by the <span class="math-container">$\frac{1}{m}$</span> factor until <span class="math-container">$dW,db$</span> rather than "absorbing" it into <span class="math-container">$dZ^{[2]}$</span>. That is, the <span class="math-container">$dZ^{[2]}$</span> term is <em>defined</em> in a way that it does not have <span class="math-container">$\frac{1}{m}$</span>. </p>
  
  <p>Observe, if you move the <span class="math-container">$\frac{1}{m}$</span> factor to be in the definition of <span class="math-container">$dZ^{[2]}$</span> and remove it from the definitions of <span class="math-container">$dW,db$</span> you will still come out with the same values for all <span class="math-container">$dW,db$</span>.</p>
  
  <p><strong>Speculation</strong></p>
  
  <p>This "deferred" multiplication might have to do with numerical stability. Or simply a stylistic choice made by A. Ng. This might also prevent one from "accidentally" multiplying by <span class="math-container">$\frac{1}{m}$</span> more than once. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-19T04:56:49.620Z
Answers.Text: Thanks for clearing the confusion. Can you please let me know where I can read this numerical stability thing and why following my approach (list in image 1 ) causes numerical stability ?
Answers.UserDisplayName: learner
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-19T23:43:48.897Z
Answers.Text: @user110244 I was only speculating as to why A. Ng presents back prop in this way. To better understand numerical stability I'd recommend searching around on the math stack exchange - surely there will be an answer better than any I could provide here.
Answers.UserDisplayName: respectful
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-19T01:56:29.283Z
Answers.LastActivityDate: 2020-03-19T02:06:31.893Z
Answers.LastEditDate: 2020-03-19T02:06:31.893Z
Answers.UserDisplayName: respectful
Body: <p>I am using the cross-entropy cost function to calculate its derivatives using different variables <span class="math-container">$Z, W$</span> and <span class="math-container">$b$</span> at different instances. Please refer image below for calculation.
  <a href="https://i.sstatic.net/C4dCN.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/C4dCN.jpg" alt="enter image description here"></a></p>
  
  <p>As per my knowledge, my derivation is correct for <span class="math-container">$dZ, dW, db$</span> and <span class="math-container">$dA$</span>, but, if I refer to Andrew Ng Coursera stuff, then I am seeing an extra <span class="math-container">$\frac{1}{m}$</span> for <span class="math-container">$dW$</span> and <span class="math-container">$db$</span>, whereas no <span class="math-container">$\frac{1}{m}$</span> in <span class="math-container">$dZ$</span>. Andrew's slides on the left represent derivative and whereas the right side of slides shows NumPy implementation corresponding to the right side equation.</p>
  
  <p><a href="https://i.sstatic.net/W6S7Y.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/W6S7Y.png" alt="enter image description here"></a>
  Can someone please explain why there is:</p>
  
  <p>1) <span class="math-container">$\frac{1}{m}$</span> in <span class="math-container">$dW^{[2]}$</span> and <span class="math-container">$db^{[2]}$</span> in Andrew's slides in NumPy representation</p>
  
  <p>2) missing <span class="math-container">$\frac{1}{m}$</span> for <span class="math-container">$dZ^{[2]}$</span> in Andrew's slides in both normal and NumPy representation.</p>
  
  <p>Am I missing something or doing it in the wrong way?</p>
  
Tags: <math><backpropagation><derivative><numpy>
Title: Why is my derivation of the back-propagation equations inconsistent with Andrew Ng's slides from Coursera?
UserDisplayName: learner
ThreadText: # Why is my derivation of the back-propagation equations inconsistent with Andrew Ng's slides from Coursera?
  
  (Asked by: username_0 on 2020-03-18)
  
  I am using the cross-entropy cost function to calculate its derivatives using different variables $Z, W$ and $b$ at different instances. Please refer image below for calculation.
  [![enter image description here](https://i.sstatic.net/C4dCN.jpg)](https://i.sstatic.net/C4dCN.jpg)
  
  As per my knowledge, my derivation is correct for $dZ, dW, db$ and $dA$, but, if I refer to Andrew Ng Coursera stuff, then I am seeing an extra $\frac{1}{m}$ for $dW$ and $db$, whereas no $\frac{1}{m}$ in $dZ$. Andrew's slides on the left represent derivative and whereas the right side of slides shows NumPy implementation corresponding to the right side equation.
  
  [![enter image description here](https://i.sstatic.net/W6S7Y.png)](https://i.sstatic.net/W6S7Y.png)
  Can someone please explain why there is:
  
  1) $\frac{1}{m}$ in $dW^{[2]}$ and $db^{[2]}$ in Andrew's slides in NumPy representation
  
  2) missing $\frac{1}{m}$ for $dZ^{[2]}$ in Andrew's slides in both normal and NumPy representation.
  
  Am I missing something or doing it in the wrong way?
  
  **Tags:** math, backpropagation, derivative, numpy
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-03-19. Score: 1
  
  **TL;DR**: This has to do with the way A. Ng has defined back propagation for the course.
  
  **Left Column**
  
  This is only with respect to *one* input example and so the $\frac{1}{m}$ factor reduces to 1 and can be omitted. He uses lower case to represent one input example (eg a vector $dz$) and upper case with respect to a (mini-)batch (eg a matrix $dZ$).
  
  **The $\frac{1}{m}$ factors in $dW,db$**
  
  In this definition of backprop, he "defers" multiplying by the $\frac{1}{m}$ factor until $dW,db$ rather than "absorbing" it into $dZ^{[2]}$. That is, the $dZ^{[2]}$ term is *defined* in a way that it does not have $\frac{1}{m}$.
  
  Observe, if you move the $\frac{1}{m}$ factor to be in the definition of $dZ^{[2]}$ and remove it from the definitions of $dW,db$ you will still come out with the same values for all $dW,db$.
  
  **Speculation**
  
  This "deferred" multiplication might have to do with numerical stability. Or simply a stylistic choice made by A. Ng. This might also prevent one from "accidentally" multiplying by $\frac{1}{m}$ more than once.
  
  > **Comments:**
  > **username_0**: Thanks for clearing the confusion. Can you please let me know where I can read this numerical stability thing and why following my approach (list in image 1 ) causes numerical stability ?  
  > **username_1**: @user110244 I was only speculating as to why A. Ng presents back prop in this way. To better understand numerical stability I'd recommend searching around on the math stack exchange - surely there will be an answer better than any I could provide here.  
  
  ---
  
  