Answers.Body: <p>In general a cost function can be negative. The more negative, the better of course, because you are measuring a cost the objective is to minimise it.</p>
  
  <p>A standard Mean Squared Error function cannot be negative. The lowest possible value is <span class="math-container">$0$</span>, when there is no output error from any example input.</p>
  
  <blockquote>
    <p>How can our cost function which is mean squared error have a value under 0?</p>
  </blockquote>
  
  <p>It cannot. You don't link the precise graph or lecture where you saw this, but I would suspect Andrew Ng drew a representative graph for <em>any</em> cost function in order to point out that it would typically have an optimum, minimum value. He may have been talking at the same time about MSE as an example.</p>
  
  <p>Many loss or cost functions are designed with an absolute minimum of <span class="math-container">$0$</span> possible for "no error" results. In supervised learning that is often a simple consequence of basing the cost on the difference between the model outputs and desired outputs. So in supervised learning problems of regression and classification, you will rarely see a negative cost function value. But there is no absolute rule against negative costs in principle.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-17T15:56:55.027Z
Answers.LastActivityDate: 2018-11-17T16:21:09.437Z
Answers.LastEditDate: 2018-11-17T16:21:09.437Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Mean squared error terms must be positive because the square of a number is positive.   Therefore the sum (cost) is positive.  The error is the difference between the hypothesis and the observation.</p>
  
  <p>I would focus on understanding why Ng seeks to minimize J and how it is minimization is achieved with partial derivatives via matrix implementation.  </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-17T18:48:46.827Z
Answers.LastActivityDate: 2018-11-17T22:29:18.093Z
Answers.LastEditDate: 2018-11-17T22:29:18.093Z
Answers.UserDisplayName: gatorback
Answers.Body: <p>I would like to add more information to the <a href="https://ai.stackexchange.com/a/9023/59824">@Neil Slater's answer</a>.</p>
  <p>Andrew sir has used a &quot;generic&quot; graph to show min loss function mean higher performance. That is not always the case, sometimes its over fitting. Let's no go into that realm of machine learning I am not that pro</p>
  <p>The output of the function in maths is called <code>range</code> and input is called <code>domain</code>. We are using MSE in the linear regression because it is sensitive to the outliers and help in penalizing the parameters to get more accurate value that fits the training example (aka line of best fit).</p>
  <p>In the MSE, the minimum loss would be <span class="math-container">$\hat y$</span> = y which means the difference is <span class="math-container">$0$</span>, so does the mean <span class="math-container">$\mu$</span> would be zero. Therefore the range of function  would be <span class="math-container">$[0, \infty)$</span>. In case of MSE, the min value of the cost function would be <span class="math-container">$0$</span> no matter what.</p>
  <p>Do not confuse the <span class="math-container">$0$</span> min value of the MSE with the min <span class="math-container">$0$</span> of gradient descent of the cost function. Remember, in the gradient descent there is no 2 in the power, it can be negative, despite being positive value of the MSE function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-11T09:35:03.050Z
Answers.LastActivityDate: 2023-01-11T09:35:03.050Z
Answers.UserDisplayName: tbhaxor
Answers.Body: <p>I am watching the same course too, and I think that in the example graph, the cost function is not a sum of MSE (Mean squarred errors), but it could be a cubic one, so a sum of cubical errors, and thus the cost function could be negative: as there are a variety of cost functions, the MSE ones are not adapted for every problems, and other formulations could work better.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-08T16:32:30.303Z
Answers.LastActivityDate: 2020-01-09T08:13:14.300Z
Answers.LastEditDate: 2020-01-09T08:13:14.300Z
Answers.UserDisplayName: Meriem Slimani
Body: <p>I'm new to machine learning. I was watching a Prof. Andrew Ng's video about gradient descent from the machine learning online course. It said that we want our cost function (in this case, the <em>mean squared error</em>) to have the minimum value, but that minimum value shown in the graph was not 0. It was a negative number! </p>
  
  <p>How can our cost function, which is mean squared error, have a negative value, given that the square of a real number is always positive? Even if it is possible, don't we want our error to be 0?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-09T06:16:34.900Z
Comments.Text: If you use quantum probabilities :) https://en.wikipedia.org/wiki/Quantum_probability
Comments.UserDisplayName: mirror2image
Tags: <machine-learning><gradient-descent><objective-functions>
Title: Can the mean squared error be negative?
UserDisplayName: Borna Ghahnoosh
ThreadText: # Can the mean squared error be negative?
  
  (Asked by: username_0 on 2018-11-17)
  
  I'm new to machine learning. I was watching a Prof. Andrew Ng's video about gradient descent from the machine learning online course. It said that we want our cost function (in this case, the *mean squared error*) to have the minimum value, but that minimum value shown in the graph was not 0. It was a negative number!
  
  How can our cost function, which is mean squared error, have a negative value, given that the square of a real number is always positive? Even if it is possible, don't we want our error to be 0?
  
  **Tags:** machine-learning, gradient-descent, objective-functions
  > **Comments:**
  > **username_1**: If you use quantum probabilities :) https://en.wikipedia.org/wiki/Quantum_probability  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_2 on 2018-11-17. Score: 10
  
  In general a cost function can be negative. The more negative, the better of course, because you are measuring a cost the objective is to minimise it.
  
  A standard Mean Squared Error function cannot be negative. The lowest possible value is $0$, when there is no output error from any example input.
  
  > How can our cost function which is mean squared error have a value under 0?
  
  It cannot. You don't link the precise graph or lecture where you saw this, but I would suspect Andrew Ng drew a representative graph for *any* cost function in order to point out that it would typically have an optimum, minimum value. He may have been talking at the same time about MSE as an example.
  
  Many loss or cost functions are designed with an absolute minimum of $0$ possible for "no error" results. In supervised learning that is often a simple consequence of basing the cost on the difference between the model outputs and desired outputs. So in supervised learning problems of regression and classification, you will rarely see a negative cost function value. But there is no absolute rule against negative costs in principle.
  
  
  ---
  
  ### Answer by username_3 on 2018-11-17. Score: 1
  
  Mean squared error terms must be positive because the square of a number is positive. Therefore the sum (cost) is positive. The error is the difference between the hypothesis and the observation.
  
  I would focus on understanding why Ng seeks to minimize J and how it is minimization is achieved with partial derivatives via matrix implementation.
  
  
  ---
  
  ### Answer by username_4 on 2023-01-11. Score: 0
  
  I would like to add more information to the [@Neil Slater's answer](https://ai.stackexchange.com/a/9023/59824).
  
  Andrew sir has used a "generic" graph to show min loss function mean higher performance. That is not always the case, sometimes its over fitting. Let's no go into that realm of machine learning I am not that pro
  
  The output of the function in maths is called `range` and input is called `domain`. We are using MSE in the linear regression because it is sensitive to the outliers and help in penalizing the parameters to get more accurate value that fits the training example (aka line of best fit).
  
  In the MSE, the minimum loss would be $\hat y$ = y which means the difference is $0$, so does the mean $\mu$ would be zero. Therefore the range of function would be $[0, \infty)$. In case of MSE, the min value of the cost function would be $0$ no matter what.
  
  Do not confuse the $0$ min value of the MSE with the min $0$ of gradient descent of the cost function. Remember, in the gradient descent there is no 2 in the power, it can be negative, despite being positive value of the MSE function.
  
  
  ---
  
  ### Answer by username_5 on 2020-01-08. Score: 0
  
  I am watching the same course too, and I think that in the example graph, the cost function is not a sum of MSE (Mean squarred errors), but it could be a cubic one, so a sum of cubical errors, and thus the cost function could be negative: as there are a variety of cost functions, the MSE ones are not adapted for every problems, and other formulations could work better.
  
  
  ---
  
  