<p>Mean Square Error (MSE) is a quadratic function and the further you go away from your optimum the bigger (quadratic) the MSE gets. Take <span class="math-container">$o_{expected}=20$</span> and <span class="math-container">$o_{net}=40$</span> as example. Your MSE is then 400, because
<span class="math-container">$MSE = (o_{expected}-o_{net})^2$</span>.</p>
<p>Just imagine <span class="math-container">$y = x^2$</span> with x being the output of your network.
If you want to shift the parabola with optimum at <span class="math-container">$20$</span> the formula you get is <span class="math-container">$y = (20-x)^2$</span>.
For every new case you train the  net on, you get a different parabola with different parameters.</p>

CC BY-SA 4.0
2021-04-20T06:40:03.210Z
2021-04-20T12:55:37.853Z
2021-04-20T12:55:37.853Z
SmarArror
<p>I'm looking at the loss function: mean squared error with gradient descent in machine learning. I'm building a single-neuron network (perceptron) that outputs a linear number. For example:</p>
<p>Input * Weight + Bias &gt; linear activation &gt; output.</p>
<p>Let's say the output is 40 while I expect the number 20. That means the loss function has to correct the weights+bias from 40 towards 20.</p>
<p>What I don't understand about mean squared error + gradient descent is: why is this number 40 displayed as a point on a <strong>parabola</strong>?</p>
<p>Does this parabola represent all possible outcomes? Why isn't it just a line? How do I know where on the parabola the point &quot;40&quot; is?</p>
<p><a href="https://i.sstatic.net/WPwui.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WPwui.png" alt="image of parabola" /></a></p>

CC BY-SA 4.0
2021-04-18T09:38:04.173Z
Are you using a specific loss function here? Please use [edit] to explain which one. I expect it is MSE, and if so that should cover everything needed to answer your question
Neil Slater
CC BY-SA 4.0
2021-04-18T10:18:21.710Z
Every minimum point can be approximated by a parabola.
Kostya
CC BY-SA 4.0
2021-04-18T12:02:20.557Z
@Kostya: Yes, but you probably would not draw a parabola for e.g. $\mathcal{L}(\hat{y}, y) = |\hat{y} - y|$
Neil Slater
CC BY-SA 4.0
2021-04-19T14:07:26.953Z
I suspect the illustration isn't meant to be taken literally; instead, I suspect the author is intending to illustrate that gradient descent attempts to solve the minimization problem by moving downward toward what is (hopefully) a global minimum on some complex surface.
David Hoelzer
CC BY-SA 4.0
2021-04-19T21:39:42.757Z
I added "mean squared error" to the question for clarity. The question is still why would all the possible "wrong" loss values happen to be points on a parabola (say, 40, 80, 14... all wrong values)... That connection is not explained in most tutorials.
Kokodoko
<objective-functions><gradient-descent>
Why is loss displayed as a parabola in mean squared error with gradient descent?
Kokodoko
# Why is loss displayed as a parabola in mean squared error with gradient descent?

(Asked by: username_0 on 2021-04-18)

I'm looking at the loss function: mean squared error with gradient descent in machine learning. I'm building a single-neuron network (perceptron) that outputs a linear number. For example:

Input \* Weight + Bias > linear activation > output.

Let's say the output is 40 while I expect the number 20. That means the loss function has to correct the weights+bias from 40 towards 20.

What I don't understand about mean squared error + gradient descent is: why is this number 40 displayed as a point on a **parabola**?

Does this parabola represent all possible outcomes? Why isn't it just a line? How do I know where on the parabola the point "40" is?

[![image of parabola](https://i.sstatic.net/WPwui.png)](https://i.sstatic.net/WPwui.png)

**Tags:** objective-functions, gradient-descent

> **Comments:**
> **username_1**: Are you using a specific loss function here? Please use [edit] to explain which one. I expect it is MSE, and if so that should cover everything needed to answer your question  
> **username_2**: Every minimum point can be approximated by a parabola.  
> **username_1**: @username_2: Yes, but you probably would not draw a parabola for e.g. $\mathcal{L}(\hat{y}, y) = |\hat{y} - y|$  
> **username_3**: I suspect the illustration isn't meant to be taken literally; instead, I suspect the author is intending to illustrate that gradient descent attempts to solve the minimization problem by moving downward toward what is (hopefully) a global minimum on some complex surface.  
> **username_0**: I added "mean squared error" to the question for clarity. The question is still why would all the possible "wrong" loss values happen to be points on a parabola (say, 40, 80, 14... all wrong values)... That connection is not explained in most tutorials.  

---

## 1 answers

### Answer by username_4 on 2021-04-20. Score: 1 | Accepted answer

Mean Square Error (MSE) is a quadratic function and the further you go away from your optimum the bigger (quadratic) the MSE gets. Take $o\_{expected}=20$ and $o\_{net}=40$ as example. Your MSE is then 400, because
$MSE = (o\_{expected}-o\_{net})^2$.

Just imagine $y = x^2$ with x being the output of your network.
If you want to shift the parabola with optimum at $20$ the formula you get is $y = (20-x)^2$.
For every new case you train the net on, you get a different parabola with different parameters.


---

