Answers.Body: <blockquote>
    <p>The MSE can be defined as <span class="math-container">$(\hat{y} - y)^2$</span>, which should be equivalent to <span class="math-container">$(y - \hat{y})^2$</span></p>
  </blockquote>
  
  <p>They are not just "equivalent". It is actually the exact same function, with two different ways to write it.</p>
  
  <p><span class="math-container">$$(\hat{y} - y)^2 = (\hat{y} - y)(\hat{y} - y) = \hat{y}^2 -2\hat{y}y + y^2$$</span></p>
  
  <p><span class="math-container">$$(y - \hat{y})^2 = (y -\hat{y})(y - \hat{y}) = y^2 -2y\hat{y} + \hat{y}^2$$</span></p>
  
  <p>These are exactly the same function. Not just "equivalent" or "equivalent everywhere", but actually the same function. It is therefore no surprise that any derivative is also the same - including the partial derivative with respect to <span class="math-container">$\hat{y}$</span> which is what you typically use to drive gradient descent.</p>
  
  <p>The two ways of writing the function is because it is a square and thus has two factorisations. When you write it as a square you can choose which form to use for the inner term.</p>
  
  <blockquote>
    <p>Which function [form] should I use to compute the gradient?</p>
  </blockquote>
  
  <p>You can use either form, it does not matter. They represent the same function and have the same gradient.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-31T15:54:08.133Z
Answers.LastActivityDate: 2019-06-01T08:14:11.513Z
Answers.LastEditDate: 2019-06-01T08:14:11.513Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>The derivative is the same as far as I understand it.  </p>
  
  <p>If <span class="math-container">$y$</span> is constant and <span class="math-container">$\hat{y}$</span> is the variable the result will be:<br>
  <span class="math-container">$((\hat{y} - y)^2)' = 2(\hat{y} - y)$</span><br>
  and for the other formula:<br>
  <span class="math-container">$((y - \hat{y})^2)' = -2(y - \hat{y})$</span><br>
  which is the same.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-31T16:09:29.643Z
Answers.Text: I can see that the second equation can only be derived if you have taken the correct route for partial derivative (I commented earlier that it looked wrong - but actually I was wrong to say that).  Usually tutorials don't consider that $y$ is a "constant", but that this is a *partial* derivative, where we only care about the gradient w.r.t. $\hat{y}$. The result is much the same, but either way it may help to show a step of expansion
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-31T13:23:16.203Z
Answers.LastActivityDate: 2019-05-31T13:23:16.203Z
Answers.UserDisplayName: Miguel Saraiva
Answers.Body: <p>The derivative of <span class="math-container">$\mathcal{L_1}(y, x) = (\hat{y} - y)^2 = (f(x) - y)^2$</span> with respect to <span class="math-container">$\hat{y}$</span>, where <span class="math-container">$f$</span> is the model and <span class="math-container">$\hat{y} = f(x)$</span> is the output of the model, is </p>
  
  <p><span class="math-container">\begin{align}
  \frac{d}{d \hat{y}} \mathcal{L_1}
  &amp;= \frac{d}{d \hat{y}} (\hat{y} - y)^2 \\
  &amp;= 2(\hat{y} - y) \frac{d}{d \hat{y}} (\hat{y} - y) \\
  &amp;= 2(\hat{y} - y) (1) \\
  &amp;= 2(\hat{y} - y) 
  \end{align}</span></p>
  
  <p>The derivative of <span class="math-container">$\mathcal{L_2}(y, x) = (y - \hat{y})^2 = (y - f(x))^2$</span> w.r.t <span class="math-container">$\hat{y}$</span> is</p>
  
  <p><span class="math-container">\begin{align}
  \frac{d}{d \hat{y}} \mathcal{L_2}
  &amp;= \frac{d}{d \hat{y}} (y - \hat{y})^2 \\
  &amp;= 2(y -\hat{y}) \frac{d}{d \hat{y}} (y -\hat{y}) \\
  &amp;= 2(y - \hat{y})(-1)\\
  &amp;= -2(y - \hat{y})\\
  &amp;= 2(\hat{y} - y)
  \end{align}</span></p>
  
  <p>So, the derivatives of <span class="math-container">$\mathcal{L_1}$</span> and <span class="math-container">$\mathcal{L_2}$</span> are the same.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-31T13:27:46.253Z
Answers.LastActivityDate: 2019-06-01T23:03:10.153Z
Answers.LastEditDate: 2019-06-01T23:03:10.153Z
Answers.UserDisplayName: nbro
Body: <p>The MSE can be defined as <span class="math-container">$(\hat{y} - y)^2$</span>, which should be equal to <span class="math-container">$(y - \hat{y})^2$</span>, but I think their derivative is different, so I am confused of what derivative will I use for computing my gradient. Can someone explain for me what term to use?</p>
  
Tags: <math><gradient-descent><objective-functions>
Title: Which function $(\hat{y} - y)^2$ or $(y - \hat{y})^2$ should I use to compute the gradient?
UserDisplayName: Jerico Bayod
ThreadText: # Which function $(\hat{y} - y)^2$ or $(y - \hat{y})^2$ should I use to compute the gradient?
  
  (Asked by: username_0 on 2019-05-31)
  
  The MSE can be defined as $(\hat{y} - y)^2$, which should be equal to $(y - \hat{y})^2$, but I think their derivative is different, so I am confused of what derivative will I use for computing my gradient. Can someone explain for me what term to use?
  
  **Tags:** math, gradient-descent, objective-functions
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2019-05-31. Score: 7 | Accepted answer
  
  The derivative of $\mathcal{L\_1}(y, x) = (\hat{y} - y)^2 = (f(x) - y)^2$ with respect to $\hat{y}$, where $f$ is the model and $\hat{y} = f(x)$ is the output of the model, is
  
  \begin{align}
  \frac{d}{d \hat{y}} \mathcal{L\_1}
  &= \frac{d}{d \hat{y}} (\hat{y} - y)^2 \\
  &= 2(\hat{y} - y) \frac{d}{d \hat{y}} (\hat{y} - y) \\
  &= 2(\hat{y} - y) (1) \\
  &= 2(\hat{y} - y)
  \end{align}
  
  The derivative of $\mathcal{L\_2}(y, x) = (y - \hat{y})^2 = (y - f(x))^2$ w.r.t $\hat{y}$ is
  
  \begin{align}
  \frac{d}{d \hat{y}} \mathcal{L\_2}
  &= \frac{d}{d \hat{y}} (y - \hat{y})^2 \\
  &= 2(y -\hat{y}) \frac{d}{d \hat{y}} (y -\hat{y}) \\
  &= 2(y - \hat{y})(-1)\\
  &= -2(y - \hat{y})\\
  &= 2(\hat{y} - y)
  \end{align}
  
  So, the derivatives of $\mathcal{L\_1}$ and $\mathcal{L\_2}$ are the same.
  
  
  ---
  
  ### Answer by username_1 on 2019-05-31. Score: 7
  
  > The MSE can be defined as $(\hat{y} - y)^2$, which should be equivalent to $(y - \hat{y})^2$
  
  They are not just "equivalent". It is actually the exact same function, with two different ways to write it.
  
  $$(\hat{y} - y)^2 = (\hat{y} - y)(\hat{y} - y) = \hat{y}^2 -2\hat{y}y + y^2$$
  
  $$(y - \hat{y})^2 = (y -\hat{y})(y - \hat{y}) = y^2 -2y\hat{y} + \hat{y}^2$$
  
  These are exactly the same function. Not just "equivalent" or "equivalent everywhere", but actually the same function. It is therefore no surprise that any derivative is also the same - including the partial derivative with respect to $\hat{y}$ which is what you typically use to drive gradient descent.
  
  The two ways of writing the function is because it is a square and thus has two factorisations. When you write it as a square you can choose which form to use for the inner term.
  
  > Which function [form] should I use to compute the gradient?
  
  You can use either form, it does not matter. They represent the same function and have the same gradient.
  
  
  ---
  
  ### Answer by username_2 on 2019-05-31. Score: 2
  
  The derivative is the same as far as I understand it.
  
  If $y$ is constant and $\hat{y}$ is the variable the result will be:  
  $((\hat{y} - y)^2)' = 2(\hat{y} - y)$  
  and for the other formula:  
  $((y - \hat{y})^2)' = -2(y - \hat{y})$  
  which is the same.
  
  > **Comments:**
  > **username_1**: I can see that the second equation can only be derived if you have taken the correct route for partial derivative (I commented earlier that it looked wrong - but actually I was wrong to say that).  Usually tutorials don't consider that $y$ is a "constant", but that this is a *partial* derivative, where we only care about the gradient w.r.t. $\hat{y}$. The result is much the same, but either way it may help to show a step of expansion  
  
  ---
  
  