Answers.Body: <p>What you stated looks correct :-</p>
  <p>The SSE loss function includes the summation of the errors in the other training examples of the mini-batch (each training example's loss in the mini-batch is summed for one big loss)
  SSE = Summation of (y-y')squared for all samples in mini-batch</p>
  <p>The MSE loss function averages the loss of each individual training example in the mini-batch, and then all those losses are averaged based on the mini-batch size.
  MSE = (1/m)SSE</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-31T12:36:21.367Z
Answers.Text: You can format the formulas and math symbols in this answer with latex. I would recommend that you do that.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-29T16:57:36.027Z
Answers.LastActivityDate: 2022-01-29T16:57:36.027Z
Answers.UserDisplayName: Vivek Singhal
Body: <p>I've been trying to implement my own neural network library and have been wondering if:</p>
  <ul>
  <li><p>The SSE loss function includes the summation of the errors in the other training examples of the mini-batch (each training example's loss in the mini-batch is summed for one big loss)</p>
  </li>
  <li><p>The MSE loss function averages the loss of each individual training example in the mini-batch, and then all those losses are averaged based on the mini-batch size.</p>
  </li>
  </ul>
  <p>or if the summing or averaging of the weight gradients has nothing to do with the loss function used?</p>
  <p>I feel like the answer would be clear if I knew other loss functions, if it did matter that would mean the weight gradients should be averaged for MSE and summed for SSE in mini-batch gradient descent?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-28T15:51:32.967Z
Comments.Text: seems at least a partial dup of [this](https://ai.stackexchange.com/q/20377/2444). let me know.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-28T16:03:31.710Z
Comments.Text: hi, I've seen that one before. I don't think it answers my question.
Comments.UserDisplayName: jake_prentice
Tags: <neural-networks><objective-functions><mini-batch-gradient-descent>
Title: Does the summing or averaging of the weight gradients have anything to do with the cost function used?
UserDisplayName: jake_prentice
ThreadText: # Does the summing or averaging of the weight gradients have anything to do with the cost function used?
  
  (Asked by: username_0 on 2022-01-28)
  
  I've been trying to implement my own neural network library and have been wondering if:
  
  * The SSE loss function includes the summation of the errors in the other training examples of the mini-batch (each training example's loss in the mini-batch is summed for one big loss)
  * The MSE loss function averages the loss of each individual training example in the mini-batch, and then all those losses are averaged based on the mini-batch size.
  
  or if the summing or averaging of the weight gradients has nothing to do with the loss function used?
  
  I feel like the answer would be clear if I knew other loss functions, if it did matter that would mean the weight gradients should be averaged for MSE and summed for SSE in mini-batch gradient descent?
  
  **Tags:** neural-networks, objective-functions, mini-batch-gradient-descent
  
  > **Comments:**
  > **username_1**: seems at least a partial dup of [this](https://ai.stackexchange.com/q/20377/2444). let me know.  
  > **username_0**: hi, I've seen that one before. I don't think it answers my question.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-01-29. Score: 1 | Accepted answer
  
  What you stated looks correct :-
  
  The SSE loss function includes the summation of the errors in the other training examples of the mini-batch (each training example's loss in the mini-batch is summed for one big loss)
  SSE = Summation of (y-y')squared for all samples in mini-batch
  
  The MSE loss function averages the loss of each individual training example in the mini-batch, and then all those losses are averaged based on the mini-batch size.
  MSE = (1/m)SSE
  
  > **Comments:**
  > **username_1**: You can format the formulas and math symbols in this answer with latex. I would recommend that you do that.  
  
  ---
  
  