Answers.Body: <p>Depending on what you want to do, there are advantages to other loss functions (crossentropy) and other regression models (beta regression), but there is not necessarily a reason to dislike MSE as a loss function when the target is between <span class="math-container">$0$</span> and <span class="math-container">$1$</span>, no. For instance, it might be that you know your outcome has a Gaussian distribution with a mean between <span class="math-container">$0$</span> and <span class="math-container">$1$</span>. Then what you’re doing by minimizing MSE is equivalent to maximum likelihood estimation, which has many nice statistical properties.</p>
  <p>From calculus, <span class="math-container">$f(x)$</span> and <span class="math-container">$\sqrt{f(x)}$</span> are minimized by the same <span class="math-container">$x$</span>. Thus, RMSE and MSE are equivalent as loss functions in the sense that whatever minimized one minimizes the other.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-27T10:27:43.623Z
Answers.Text: Using a cross entropy loss vs. a regression loss amount to doing two greatly different tasks. From my understand OP has output values in $[0, 1]$, whereas a cross entropy loss is used for classifications where your values(/labels) are in $\{0, 1\}$.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-27T10:33:51.463Z
Answers.Text: It might be that the true targets span the entire interval. It also could be that the targets are just the integers and the OP wants to estimate the probability. That is why it depends on the task.
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-28T02:07:29.720Z
Answers.Text: Yes just to clarify the expected output is a range between 0-1 not binary. It’s a regression problem not a classification.
Answers.UserDisplayName: Darren Rahnemoon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-28T18:13:47.507Z
Answers.Text: When aggregated over a batch of data, MSE and RMSE have different minima. And I think it is perfectly fine to use cross entropy for non-binary labels, they just represent uncertainty of the correct answer. I do this regularly with binary image masks with blurred edges.
Answers.UserDisplayName: NikoNyrh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-27T01:38:40.257Z
Answers.LastActivityDate: 2023-03-27T01:38:40.257Z
Answers.UserDisplayName: Dave
Answers.Body: <p>To summarize all the comments and what my original mistake was:</p>
  <p>The sensitivity effect is still there even though yes the squared of the error value less than 0 is less that itself however:</p>
  <pre><code>Example: 
  
  For two error values of 0.9 and 0.2
  
  MAE: 0.9
  MSE: 0.9 ^ 2 = 0.81
  
  MAE: 0.2
  MSE: 0.2 ^ 2 = 0.04
  
  MAE: 0.9 / 0.2 = 4.5 # less sensitive in MAE
  MSE: 0.81 / 0.04 = 20.25 # still significantly sensitive in MSE
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-30T20:13:28.110Z
Answers.LastActivityDate: 2023-03-30T20:13:28.110Z
Answers.UserDisplayName: Darren Rahnemoon
Body: <p>In a NN regression problem, considering that MSE is squaring the error and the error is between 0 and 1 would it be pointless to use MSE as our loss function during model training?</p>
  <p>For example:</p>
  <pre><code>MSE = (y_pred - y_true) ^ 2 
  
  @ Expected model output range [0, 1]:
  MSE = (0.1 - 0.01) ^ 2 = 0.0081
  
  // Significantly larger error is less pronounced in the MSE output
  MSE = (0.1 - 0.0001) ^ 2 = 0.00998001
  
  
  @ Expected model output range [10, 20]:
  MSE = (10 - 12) ^ 2 = 4
  
  // Significantly larger error is more pronounced in the MSE output
  MSE = (10 - 20) ^ 2 = 100
  </code></pre>
  <p>If it’s indeed useless for that range, would using RMSE allow us to use this loss function at 0-1 range to benefit from its outlier sensitivity during training or is there another loss that would mimic the effect of MSE for values between 0 and 1?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-29T21:30:48.807Z
Comments.Text: @user253751 under the assumption that since squaring error values that are <1 has the opposite effect (i.e 0.2^2 = 0.04 but 2^2 = 4) since this is being used in a loss function from which the neural network is computing gradients the outliers move the gradient less than values within the common range of errors.
Comments.UserDisplayName: Darren Rahnemoon
Tags: <deep-learning><loss><mean-squared-error>
Title: Does MSE loss function work in NN training for predicting values between 0-1?
UserDisplayName: Darren Rahnemoon
ThreadText: # Does MSE loss function work in NN training for predicting values between 0-1?
  
  (Asked by: username_0 on 2023-03-27)
  
  In a NN regression problem, considering that MSE is squaring the error and the error is between 0 and 1 would it be pointless to use MSE as our loss function during model training?
  
  For example:
  
  ```
  MSE = (y_pred - y_true) ^ 2 
  
  @ Expected model output range [0, 1]:
  MSE = (0.1 - 0.01) ^ 2 = 0.0081
  
  // Significantly larger error is less pronounced in the MSE output
  MSE = (0.1 - 0.0001) ^ 2 = 0.00998001
  
  @ Expected model output range [10, 20]:
  MSE = (10 - 12) ^ 2 = 4
  
  // Significantly larger error is more pronounced in the MSE output
  MSE = (10 - 20) ^ 2 = 100
  
  ```
  
  If it’s indeed useless for that range, would using RMSE allow us to use this loss function at 0-1 range to benefit from its outlier sensitivity during training or is there another loss that would mimic the effect of MSE for values between 0 and 1?
  
  **Tags:** deep-learning, loss, mean-squared-error
  
  > **Comments:**
  > **username_0**: @user253751 under the assumption that since squaring error values that are <1 has the opposite effect (i.e 0.2^2 = 0.04 but 2^2 = 4) since this is being used in a loss function from which the neural network is computing gradients the outliers move the gradient less than values within the common range of errors.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-03-27. Score: 0 | Accepted answer
  
  Depending on what you want to do, there are advantages to other loss functions (crossentropy) and other regression models (beta regression), but there is not necessarily a reason to dislike MSE as a loss function when the target is between $0$ and $1$, no. For instance, it might be that you know your outcome has a Gaussian distribution with a mean between $0$ and $1$. Then what you’re doing by minimizing MSE is equivalent to maximum likelihood estimation, which has many nice statistical properties.
  
  From calculus, $f(x)$ and $\sqrt{f(x)}$ are minimized by the same $x$. Thus, RMSE and MSE are equivalent as loss functions in the sense that whatever minimized one minimizes the other.
  
  > **Comments:**
  > **username_2**: Using a cross entropy loss vs. a regression loss amount to doing two greatly different tasks. From my understand OP has output values in $[0, 1]$, whereas a cross entropy loss is used for classifications where your values(/labels) are in $\{0, 1\}$.  
  > **username_1**: It might be that the true targets span the entire interval. It also could be that the targets are just the integers and the OP wants to estimate the probability. That is why it depends on the task.  
  > **username_0**: Yes just to clarify the expected output is a range between 0-1 not binary. It’s a regression problem not a classification.  
  > **username_3**: When aggregated over a batch of data, MSE and RMSE have different minima. And I think it is perfectly fine to use cross entropy for non-binary labels, they just represent uncertainty of the correct answer. I do this regularly with binary image masks with blurred edges.  
  
  ---
  
  ### Answer by username_0 on 2023-03-30. Score: 0
  
  To summarize all the comments and what my original mistake was:
  
  The sensitivity effect is still there even though yes the squared of the error value less than 0 is less that itself however:
  
  ```
  Example: 
  
  For two error values of 0.9 and 0.2
  
  MAE: 0.9
  MSE: 0.9 ^ 2 = 0.81
  
  MAE: 0.2
  MSE: 0.2 ^ 2 = 0.04
  
  MAE: 0.9 / 0.2 = 4.5 # less sensitive in MAE
  MSE: 0.81 / 0.04 = 20.25 # still significantly sensitive in MSE
  
  ```
  
  
  ---
  
  