Answers.Body: <blockquote>
  <p>If <span class="math-container">$t^i - o^i$</span> is negative, doesn't the power of 2 eliminate any negative result?</p>
  </blockquote>
  <p>In the loss function, yes that is correct, and is what you want - a measurement that gets higher due to any difference between the predicted and correct results. Minimising the value of that measurement is a goal for the optimiser.</p>
  <blockquote>
  <p>How can then exist any negative gradient at the output?</p>
  </blockquote>
  <p>The gradient of the cost function is different to the cost function. Technically it is the derivative of the cost function with respect to the parameters you can change. Taking the derivative changes the powers in the equation.</p>
  <p>If you plot the graph from a single example, you can clearly see the gradient is negative when the prediction is too low, and positive when the prediction is too high:</p>
  <p><a href="https://i.sstatic.net/8vkab.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8vkab.png" alt="SSE loss function example graph" /></a></p>
  <p>Note this is the opposite relationship to the one you suggest in the question. That is why it is called gradient <em>descent</em> when we apply corrections. The steps to correct the parameters are in the <em>opposite</em> direction to the gradient.</p>
  <p>In terms of algebra (and moving the example index into a subscript so that is doesn't get in the way visually), assuming <span class="math-container">$o_i$</span> is your output value, and <span class="math-container">$t_i$</span> is the target value, then you want the gradient of your loss function <span class="math-container">$L$</span> with respect to <span class="math-container">$o_i$</span> or <span class="math-container">$\frac{\partial L}{\partial o_i}$</span>:</p>
  <p><span class="math-container">$$\frac{\partial L}{\partial o_i} =  \frac{\partial}{\partial o_i} \frac{1}{2}(t_i - o_i)^2 = \frac{\partial}{\partial o_i} \frac{1}{2}(t_i^2 - 2t_i o_i + o_i^2) = \frac{1}{2}(-2t_i + 2o_i) = o_i - t_i$$</span></p>
  <p>This simple gradient value is negative if your output <span class="math-container">$o_i$</span> is lower than the target <span class="math-container">$t_i$</span>, and positive if <span class="math-container">$o_i$</span>is greater than <span class="math-container">$t_i$</span>.</p>
  <p>As an aside, if you wondered why the multiplier of <span class="math-container">$\frac{1}{2}$</span> was included in the definition of SSE, it is intended by design to end up with this simple result. Without it, everything works, just you would end up with the gradient being <span class="math-container">$2(o_i - t_i)$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-25T11:49:50.190Z
Answers.Text: Nice, thanks for the clear and detailed answer. Just one more question: While the SSE is the sum of the errors of all samples, the MSE divides through the number of samples to get the mean error. But: The MSE makes far more sense in a NN, right? I mean, why should the error, and so the adjustment of the weights grow with the number of samples?
Answers.UserDisplayName: Ben
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-25T17:17:05.453Z
Answers.Text: @Ben: In brief, the MSE is more used in stats-based learning algorithms than SSE because it estimates the loss for a single "typical" point, and the more data you get, the better the estimate. That means you can do things such as take a small subset (mini-batch) of your full data set, measure the loss, calculate the gradient and make a step - all the while using self-consistent (but noisy) measures of the errors and gradients. Also you can make comparisons easily between different sizes of data set. If that is not clear, please ask a new separate question.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-25T09:11:17.310Z
Answers.LastActivityDate: 2021-03-10T17:55:41.003Z
Answers.LastEditDate: 2021-03-10T17:55:41.003Z
Answers.UserDisplayName: Neil Slater
Body: <p>The formula for the <em>sum of squared errors</em> (SSE) is:</p>
  <p><span class="math-container">$$
  \frac{1}{2} \sum_{i=1}^n (t^i - o^i)^2
  $$</span></p>
  <p>I have a few related questions.</p>
  <ol>
  <li><p>If <span class="math-container">$t^i - o^i$</span> is negative, doesn't the power of 2 eliminate any negative result?</p>
  </li>
  <li><p>How can then exist any negative gradient at the output?</p>
  </li>
  <li><p>What if the output is too high, and so the gradient should be negative, meaning that the weights (on average) have to be decreased? How can it be differentiated between 'output too high' and 'output too low'?</p>
  </li>
  </ol>
  
Tags: <machine-learning><objective-functions><gradient><sse>
Title: How can the sum of squared errors have negative gradient if it's defined as the squared of the error?
UserDisplayName: Ben
ThreadText: # How can the sum of squared errors have negative gradient if it's defined as the squared of the error?
  
  (Asked by: username_0 on 2018-11-24)
  
  The formula for the *sum of squared errors* (SSE) is:
  
  $$
  \frac{1}{2} \sum\_{i=1}^n (t^i - o^i)^2
  $$
  
  I have a few related questions.
  
  1. If $t^i - o^i$ is negative, doesn't the power of 2 eliminate any negative result?
  2. How can then exist any negative gradient at the output?
  3. What if the output is too high, and so the gradient should be negative, meaning that the weights (on average) have to be decreased? How can it be differentiated between 'output too high' and 'output too low'?
  
  **Tags:** machine-learning, objective-functions, gradient, sse
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-11-25. Score: 3 | Accepted answer
  
  > If $t^i - o^i$ is negative, doesn't the power of 2 eliminate any negative result?
  
  In the loss function, yes that is correct, and is what you want - a measurement that gets higher due to any difference between the predicted and correct results. Minimising the value of that measurement is a goal for the optimiser.
  
  > How can then exist any negative gradient at the output?
  
  The gradient of the cost function is different to the cost function. Technically it is the derivative of the cost function with respect to the parameters you can change. Taking the derivative changes the powers in the equation.
  
  If you plot the graph from a single example, you can clearly see the gradient is negative when the prediction is too low, and positive when the prediction is too high:
  
  [![SSE loss function example graph](https://i.sstatic.net/8vkab.png)](https://i.sstatic.net/8vkab.png)
  
  Note this is the opposite relationship to the one you suggest in the question. That is why it is called gradient *descent* when we apply corrections. The steps to correct the parameters are in the *opposite* direction to the gradient.
  
  In terms of algebra (and moving the example index into a subscript so that is doesn't get in the way visually), assuming $o\_i$ is your output value, and $t\_i$ is the target value, then you want the gradient of your loss function $L$ with respect to $o\_i$ or $\frac{\partial L}{\partial o\_i}$:
  
  $$\frac{\partial L}{\partial o\_i} = \frac{\partial}{\partial o\_i} \frac{1}{2}(t\_i - o\_i)^2 = \frac{\partial}{\partial o\_i} \frac{1}{2}(t\_i^2 - 2t\_i o\_i + o\_i^2) = \frac{1}{2}(-2t\_i + 2o\_i) = o\_i - t\_i$$
  
  This simple gradient value is negative if your output $o\_i$ is lower than the target $t\_i$, and positive if $o\_i$is greater than $t\_i$.
  
  As an aside, if you wondered why the multiplier of $\frac{1}{2}$ was included in the definition of SSE, it is intended by design to end up with this simple result. Without it, everything works, just you would end up with the gradient being $2(o\_i - t\_i)$
  
  > **Comments:**
  > **username_0**: Nice, thanks for the clear and detailed answer. Just one more question: While the SSE is the sum of the errors of all samples, the MSE divides through the number of samples to get the mean error. But: The MSE makes far more sense in a NN, right? I mean, why should the error, and so the adjustment of the weights grow with the number of samples?  
  > **username_1**: @username_0: In brief, the MSE is more used in stats-based learning algorithms than SSE because it estimates the loss for a single "typical" point, and the more data you get, the better the estimate. That means you can do things such as take a small subset (mini-batch) of your full data set, measure the loss, calculate the gradient and make a step - all the while using self-consistent (but noisy) measures of the errors and gradients. Also you can make comparisons easily between different sizes of data set. If that is not clear, please ask a new separate question.  
  
  ---
  
  