Answers.Body: <p>Mean Square Error (MSE) is a quadratic function and the further you go away from your optimum the bigger (quadratic) the MSE gets. Take <span class="math-container">$o_{expected}=20$</span> and <span class="math-container">$o_{net}=40$</span> as example. Your MSE is then 400, because
  <span class="math-container">$MSE = (o_{expected}-o_{net})^2$</span>.</p>
  <p>Just imagine <span class="math-container">$y = x^2$</span> with x being the output of your network.
  If you want to shift the parabola with optimum at <span class="math-container">$20$</span> the formula you get is <span class="math-container">$y = (20-x)^2$</span>.
  For every new case you train the  net on, you get a different parabola with different parameters.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-20T06:40:03.210Z
Answers.LastActivityDate: 2021-04-20T12:55:37.853Z
Answers.LastEditDate: 2021-04-20T12:55:37.853Z
Answers.UserDisplayName: SmarArror
Body: <p>I'm looking at the loss function: mean squared error with gradient descent in machine learning. I'm building a single-neuron network (perceptron) that outputs a linear number. For example:</p>
  <p>Input * Weight + Bias &gt; linear activation &gt; output.</p>
  <p>Let's say the output is 40 while I expect the number 20. That means the loss function has to correct the weights+bias from 40 towards 20.</p>
  <p>What I don't understand about mean squared error + gradient descent is: why is this number 40 displayed as a point on a <strong>parabola</strong>?</p>
  <p>Does this parabola represent all possible outcomes? Why isn't it just a line? How do I know where on the parabola the point &quot;40&quot; is?</p>
  <p><a href="https://i.sstatic.net/WPwui.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WPwui.png" alt="image of parabola" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-18T09:38:04.173Z
Comments.Text: Are you using a specific loss function here? Please use [edit] to explain which one. I expect it is MSE, and if so that should cover everything needed to answer your question
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-18T10:18:21.710Z
Comments.Text: Every minimum point can be approximated by a parabola.
Comments.UserDisplayName: Kostya
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-18T12:02:20.557Z
Comments.Text: @Kostya: Yes, but you probably would not draw a parabola for e.g. $\mathcal{L}(\hat{y}, y) = |\hat{y} - y|$
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-19T14:07:26.953Z
Comments.Text: I suspect the illustration isn't meant to be taken literally; instead, I suspect the author is intending to illustrate that gradient descent attempts to solve the minimization problem by moving downward toward what is (hopefully) a global minimum on some complex surface.
Comments.UserDisplayName: David Hoelzer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-19T21:39:42.757Z
Comments.Text: I added "mean squared error" to the question for clarity. The question is still why would all the possible "wrong" loss values happen to be points on a parabola (say, 40, 80, 14... all wrong values)... That connection is not explained in most tutorials.
Comments.UserDisplayName: Kokodoko
Tags: <objective-functions><gradient-descent>
Title: Why is loss displayed as a parabola in mean squared error with gradient descent?
UserDisplayName: Kokodoko
ThreadText: # Why is loss displayed as a parabola in mean squared error with gradient descent?
  
  (Asked by: username_0 on 2021-04-18)
  
  I'm looking at the loss function: mean squared error with gradient descent in machine learning. I'm building a single-neuron network (perceptron) that outputs a linear number. For example:
  
  Input \* Weight + Bias > linear activation > output.
  
  Let's say the output is 40 while I expect the number 20. That means the loss function has to correct the weights+bias from 40 towards 20.
  
  What I don't understand about mean squared error + gradient descent is: why is this number 40 displayed as a point on a **parabola**?
  
  Does this parabola represent all possible outcomes? Why isn't it just a line? How do I know where on the parabola the point "40" is?
  
  [![image of parabola](https://i.sstatic.net/WPwui.png)](https://i.sstatic.net/WPwui.png)
  
  **Tags:** objective-functions, gradient-descent
  
  > **Comments:**
  > **username_1**: Are you using a specific loss function here? Please use [edit] to explain which one. I expect it is MSE, and if so that should cover everything needed to answer your question  
  > **username_2**: Every minimum point can be approximated by a parabola.  
  > **username_1**: @username_2: Yes, but you probably would not draw a parabola for e.g. $\mathcal{L}(\hat{y}, y) = |\hat{y} - y|$  
  > **username_3**: I suspect the illustration isn't meant to be taken literally; instead, I suspect the author is intending to illustrate that gradient descent attempts to solve the minimization problem by moving downward toward what is (hopefully) a global minimum on some complex surface.  
  > **username_0**: I added "mean squared error" to the question for clarity. The question is still why would all the possible "wrong" loss values happen to be points on a parabola (say, 40, 80, 14... all wrong values)... That connection is not explained in most tutorials.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2021-04-20. Score: 1 | Accepted answer
  
  Mean Square Error (MSE) is a quadratic function and the further you go away from your optimum the bigger (quadratic) the MSE gets. Take $o\_{expected}=20$ and $o\_{net}=40$ as example. Your MSE is then 400, because
  $MSE = (o\_{expected}-o\_{net})^2$.
  
  Just imagine $y = x^2$ with x being the output of your network.
  If you want to shift the parabola with optimum at $20$ the formula you get is $y = (20-x)^2$.
  For every new case you train the net on, you get a different parabola with different parameters.
  
  
  ---
  
  