Body: <p>In backpropagation the gradients are used to update the weights using the formula
  <span class="math-container">$$w = w - \alpha \frac{dL}{dw}$$</span>
  and the loss gradient w.r.t. weights is
  <span class="math-container">$$\frac{dL}{dw} = \frac{dL}{dz} \frac{dz}{dw} = (\frac{dL}{da} \frac{da}{dz} \frac{1}{m}) \frac{dz}{dw} $$</span></p>
  <p>Why is the there the <span class="math-container">$\frac{1}{m}$</span> term? Does batch size matter and what if it's 1?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-25T08:19:27.977Z
Comments.Text: could you post a link to the book/blogpost from which you got the formula? m should be just the total amount of weights, but the notation is not consistent across literature so to be sure I should check from your source.
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-25T22:43:40.007Z
Comments.Text: Sorry I forgot to mention that m is the number of features in the last layer's output. Here's the video link https://youtu.be/yXcQ4B-YSjQ?list=PLkDaE6sCZn6Ec-XTbcX1uRg2_u4xOEky0&t=780 but he writes it in an abbreviated way
Comments.UserDisplayName: rkuang25
Tags: <neural-networks><tensorflow><backpropagation><math><derivative>
Title: Why Is There The Term 1/m In Backpropagation
UserDisplayName: rkuang25
ThreadText: # Why Is There The Term 1/m In Backpropagation
  
  (Asked by: username_0 on 2022-07-25)
  
  In backpropagation the gradients are used to update the weights using the formula
  $$w = w - \alpha \frac{dL}{dw}$$
  and the loss gradient w.r.t. weights is
  $$\frac{dL}{dw} = \frac{dL}{dz} \frac{dz}{dw} = (\frac{dL}{da} \frac{da}{dz} \frac{1}{m}) \frac{dz}{dw} $$
  
  Why is the there the $\frac{1}{m}$ term? Does batch size matter and what if it's 1?
  
  **Tags:** neural-networks, tensorflow, backpropagation, math, derivative
  
  > **Comments:**
  > **username_1**: could you post a link to the book/blogpost from which you got the formula? m should be just the total amount of weights, but the notation is not consistent across literature so to be sure I should check from your source.  
  > **username_0**: Sorry I forgot to mention that m is the number of features in the last layer's output. Here's the video link https://youtu.be/yXcQ4B-YSjQ?list=PLkDaE6sCZn6Ec-XTbcX1uRg2_u4xOEky0&t=780 but he writes it in an abbreviated way  
  
  ---
  
  ## 0 answers
  
  