<blockquote>
  <p>The MSE can be defined as <span class="math-container">$(\hat{y} - y)^2$</span>, which should be equivalent to <span class="math-container">$(y - \hat{y})^2$</span></p>
</blockquote>

<p>They are not just "equivalent". It is actually the exact same function, with two different ways to write it.</p>

<p><span class="math-container">$$(\hat{y} - y)^2 = (\hat{y} - y)(\hat{y} - y) = \hat{y}^2 -2\hat{y}y + y^2$$</span></p>

<p><span class="math-container">$$(y - \hat{y})^2 = (y -\hat{y})(y - \hat{y}) = y^2 -2y\hat{y} + \hat{y}^2$$</span></p>

<p>These are exactly the same function. Not just "equivalent" or "equivalent everywhere", but actually the same function. It is therefore no surprise that any derivative is also the same - including the partial derivative with respect to <span class="math-container">$\hat{y}$</span> which is what you typically use to drive gradient descent.</p>

<p>The two ways of writing the function is because it is a square and thus has two factorisations. When you write it as a square you can choose which form to use for the inner term.</p>

<blockquote>
  <p>Which function [form] should I use to compute the gradient?</p>
</blockquote>

<p>You can use either form, it does not matter. They represent the same function and have the same gradient.</p>

CC BY-SA 4.0
2019-05-31T15:54:08.133Z
2019-06-01T08:14:11.513Z
2019-06-01T08:14:11.513Z
Neil Slater
<p>The derivative is the same as far as I understand it.  </p>

<p>If <span class="math-container">$y$</span> is constant and <span class="math-container">$\hat{y}$</span> is the variable the result will be:<br>
<span class="math-container">$((\hat{y} - y)^2)' = 2(\hat{y} - y)$</span><br>
and for the other formula:<br>
<span class="math-container">$((y - \hat{y})^2)' = -2(y - \hat{y})$</span><br>
which is the same.</p>

CC BY-SA 4.0
2019-05-31T16:09:29.643Z
I can see that the second equation can only be derived if you have taken the correct route for partial derivative (I commented earlier that it looked wrong - but actually I was wrong to say that).  Usually tutorials don't consider that $y$ is a "constant", but that this is a *partial* derivative, where we only care about the gradient w.r.t. $\hat{y}$. The result is much the same, but either way it may help to show a step of expansion
Neil Slater
CC BY-SA 4.0
2019-05-31T13:23:16.203Z
2019-05-31T13:23:16.203Z
Miguel Saraiva
<p>The derivative of <span class="math-container">$\mathcal{L_1}(y, x) = (\hat{y} - y)^2 = (f(x) - y)^2$</span> with respect to <span class="math-container">$\hat{y}$</span>, where <span class="math-container">$f$</span> is the model and <span class="math-container">$\hat{y} = f(x)$</span> is the output of the model, is </p>

<p><span class="math-container">\begin{align}
\frac{d}{d \hat{y}} \mathcal{L_1}
&amp;= \frac{d}{d \hat{y}} (\hat{y} - y)^2 \\
&amp;= 2(\hat{y} - y) \frac{d}{d \hat{y}} (\hat{y} - y) \\
&amp;= 2(\hat{y} - y) (1) \\
&amp;= 2(\hat{y} - y) 
\end{align}</span></p>

<p>The derivative of <span class="math-container">$\mathcal{L_2}(y, x) = (y - \hat{y})^2 = (y - f(x))^2$</span> w.r.t <span class="math-container">$\hat{y}$</span> is</p>

<p><span class="math-container">\begin{align}
\frac{d}{d \hat{y}} \mathcal{L_2}
&amp;= \frac{d}{d \hat{y}} (y - \hat{y})^2 \\
&amp;= 2(y -\hat{y}) \frac{d}{d \hat{y}} (y -\hat{y}) \\
&amp;= 2(y - \hat{y})(-1)\\
&amp;= -2(y - \hat{y})\\
&amp;= 2(\hat{y} - y)
\end{align}</span></p>

<p>So, the derivatives of <span class="math-container">$\mathcal{L_1}$</span> and <span class="math-container">$\mathcal{L_2}$</span> are the same.</p>

CC BY-SA 4.0
2019-05-31T13:27:46.253Z
2019-06-01T23:03:10.153Z
2019-06-01T23:03:10.153Z
nbro
<p>The MSE can be defined as <span class="math-container">$(\hat{y} - y)^2$</span>, which should be equal to <span class="math-container">$(y - \hat{y})^2$</span>, but I think their derivative is different, so I am confused of what derivative will I use for computing my gradient. Can someone explain for me what term to use?</p>

<math><gradient-descent><objective-functions>
Which function $(\hat{y} - y)^2$ or $(y - \hat{y})^2$ should I use to compute the gradient?
Jerico Bayod
# Which function $(\hat{y} - y)^2$ or $(y - \hat{y})^2$ should I use to compute the gradient?

(Asked by: username_0 on 2019-05-31)

The MSE can be defined as $(\hat{y} - y)^2$, which should be equal to $(y - \hat{y})^2$, but I think their derivative is different, so I am confused of what derivative will I use for computing my gradient. Can someone explain for me what term to use?

**Tags:** math, gradient-descent, objective-functions


---

## 3 answers

### Answer by username_3 on 2019-05-31. Score: 7 | Accepted answer

The derivative of $\mathcal{L\_1}(y, x) = (\hat{y} - y)^2 = (f(x) - y)^2$ with respect to $\hat{y}$, where $f$ is the model and $\hat{y} = f(x)$ is the output of the model, is

\begin{align}
\frac{d}{d \hat{y}} \mathcal{L\_1}
&= \frac{d}{d \hat{y}} (\hat{y} - y)^2 \\
&= 2(\hat{y} - y) \frac{d}{d \hat{y}} (\hat{y} - y) \\
&= 2(\hat{y} - y) (1) \\
&= 2(\hat{y} - y)
\end{align}

The derivative of $\mathcal{L\_2}(y, x) = (y - \hat{y})^2 = (y - f(x))^2$ w.r.t $\hat{y}$ is

\begin{align}
\frac{d}{d \hat{y}} \mathcal{L\_2}
&= \frac{d}{d \hat{y}} (y - \hat{y})^2 \\
&= 2(y -\hat{y}) \frac{d}{d \hat{y}} (y -\hat{y}) \\
&= 2(y - \hat{y})(-1)\\
&= -2(y - \hat{y})\\
&= 2(\hat{y} - y)
\end{align}

So, the derivatives of $\mathcal{L\_1}$ and $\mathcal{L\_2}$ are the same.


---

### Answer by username_1 on 2019-05-31. Score: 7

> The MSE can be defined as $(\hat{y} - y)^2$, which should be equivalent to $(y - \hat{y})^2$

They are not just "equivalent". It is actually the exact same function, with two different ways to write it.

$$(\hat{y} - y)^2 = (\hat{y} - y)(\hat{y} - y) = \hat{y}^2 -2\hat{y}y + y^2$$

$$(y - \hat{y})^2 = (y -\hat{y})(y - \hat{y}) = y^2 -2y\hat{y} + \hat{y}^2$$

These are exactly the same function. Not just "equivalent" or "equivalent everywhere", but actually the same function. It is therefore no surprise that any derivative is also the same - including the partial derivative with respect to $\hat{y}$ which is what you typically use to drive gradient descent.

The two ways of writing the function is because it is a square and thus has two factorisations. When you write it as a square you can choose which form to use for the inner term.

> Which function [form] should I use to compute the gradient?

You can use either form, it does not matter. They represent the same function and have the same gradient.


---

### Answer by username_2 on 2019-05-31. Score: 2

The derivative is the same as far as I understand it.

If $y$ is constant and $\hat{y}$ is the variable the result will be:  
$((\hat{y} - y)^2)' = 2(\hat{y} - y)$  
and for the other formula:  
$((y - \hat{y})^2)' = -2(y - \hat{y})$  
which is the same.

> **Comments:**
> **username_1**: I can see that the second equation can only be derived if you have taken the correct route for partial derivative (I commented earlier that it looked wrong - but actually I was wrong to say that).  Usually tutorials don't consider that $y$ is a "constant", but that this is a *partial* derivative, where we only care about the gradient w.r.t. $\hat{y}$. The result is much the same, but either way it may help to show a step of expansion  

---

