Answers.Body: <p>You can not use error to reliably measure accuracy. Error is best used as a measure of how fast the model is currently learning.</p>
  
  <p>As an example, using different loss functions (cross entorpy vs MSE) results in massively different values for the error at similar accuracy.</p>
  
  <p>Also considering this, an error of 0.0000000001 quite often has lower validation set accuracy then and error of 0.1, as the prior is likely over trained.</p>
  
  <p>As for you second question, yes this is because MSE has a huge bias towards outliers. I have personally found regression networks to struggle in most circumstances, so if it is at all possible to turn the network into a classifier, you may see an improvement.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-14T10:46:00.507Z
Answers.Text: Not entirely correct; 1) accuracy in the *technical* sense used in ML has to do with *classification* (and not regression) problems, where MSE loss would be completely meaningless 2) in regression settings, it is very common to use the loss function itself (MSE, MAE  RMSE etc) also as the "business" metric, in contrast with classification problems where this is never done.
Answers.UserDisplayName: desertnaut
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-14T00:50:39.493Z
Answers.LastActivityDate: 2019-09-14T00:50:39.493Z
Answers.UserDisplayName: Recessive
Answers.Body: <p>Just as a general remark, notice that technically we don't use the term "accuracy" for regression settings, such as yours - only for classification ones.</p>
  
  <blockquote>
    <p>If RMSE is 'in the units of the quantity being estimated', does this mean we can say: "The network is on average (1-SQRT(0.019))*100 = 86.2% accurate"? </p>
  </blockquote>
  
  <p>No.</p>
  
  <p>The advantage of the RMSE, as you have correctly quoted, is that it is in the same units with your predicted quantity; so, if this quantity is, say, USD, you can say (to the <em>business</em> user) that the error of the model is 0.019 USD, and this can be perfectly fine by itself. But you cannot convert it to a percentage - it would be meaningless.</p>
  
  <p>If required to give the performance of a <em>regression</em> model in a percentage, your best option would be the Mean Absolute Percentage Error (<a href="https://en.wikipedia.org/wiki/Mean_absolute_percentage_error" rel="nofollow noreferrer">MAPE</a>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-14T10:55:53.003Z
Answers.LastActivityDate: 2019-09-14T10:55:53.003Z
Answers.UserDisplayName: desertnaut
Body: <p>I have a regression MLP network with all input values between 0 and 1, and am using MSE for the loss function. The minimum MSE over the validation sample set comes to 0.019. So how to express the 'accuracy' of this network in 'lay' terms? If RMSE is 'in the units of the quantity being estimated', does this mean we can say: "The network is on average (1-SQRT(0.019))*100 =  86.2% accurate"? </p>
  
  <p>Also, in the validation data set, there are three 'extreme' expected values. The lowest MSE results in predicted values closer to these three values, but not as close to all the other values, whereas a slightly higher MSE results in the opposite - predicted values further from the 'extreme' values but more accurate relative to all other expected values (and this outcome is actually preferred in the case I'm dealing with). I assume this can be explained by RMSE's sensitivity to outliers? </p>
  
Tags: <neural-networks><machine-learning><regression><accuracy><mean-squared-error>
Title: How to express accuracy of a regression ANN that uses MSE loss function?
UserDisplayName: samiant
ThreadText: # How to express accuracy of a regression ANN that uses MSE loss function?
  
  (Asked by: username_0 on 2019-09-13)
  
  I have a regression MLP network with all input values between 0 and 1, and am using MSE for the loss function. The minimum MSE over the validation sample set comes to 0.019. So how to express the 'accuracy' of this network in 'lay' terms? If RMSE is 'in the units of the quantity being estimated', does this mean we can say: "The network is on average (1-SQRT(0.019))\*100 = 86.2% accurate"?
  
  Also, in the validation data set, there are three 'extreme' expected values. The lowest MSE results in predicted values closer to these three values, but not as close to all the other values, whereas a slightly higher MSE results in the opposite - predicted values further from the 'extreme' values but more accurate relative to all other expected values (and this outcome is actually preferred in the case I'm dealing with). I assume this can be explained by RMSE's sensitivity to outliers?
  
  **Tags:** neural-networks, machine-learning, regression, accuracy, mean-squared-error
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-09-14. Score: 2
  
  You can not use error to reliably measure accuracy. Error is best used as a measure of how fast the model is currently learning.
  
  As an example, using different loss functions (cross entorpy vs MSE) results in massively different values for the error at similar accuracy.
  
  Also considering this, an error of 0.0000000001 quite often has lower validation set accuracy then and error of 0.1, as the prior is likely over trained.
  
  As for you second question, yes this is because MSE has a huge bias towards outliers. I have personally found regression networks to struggle in most circumstances, so if it is at all possible to turn the network into a classifier, you may see an improvement.
  
  > **Comments:**
  > **username_2**: Not entirely correct; 1) accuracy in the *technical* sense used in ML has to do with *classification* (and not regression) problems, where MSE loss would be completely meaningless 2) in regression settings, it is very common to use the loss function itself (MSE, MAE  RMSE etc) also as the "business" metric, in contrast with classification problems where this is never done.  
  
  ---
  
  ### Answer by username_2 on 2019-09-14. Score: 1
  
  Just as a general remark, notice that technically we don't use the term "accuracy" for regression settings, such as yours - only for classification ones.
  
  > If RMSE is 'in the units of the quantity being estimated', does this mean we can say: "The network is on average (1-SQRT(0.019))\*100 = 86.2% accurate"?
  
  No.
  
  The advantage of the RMSE, as you have correctly quoted, is that it is in the same units with your predicted quantity; so, if this quantity is, say, USD, you can say (to the *business* user) that the error of the model is 0.019 USD, and this can be perfectly fine by itself. But you cannot convert it to a percentage - it would be meaningless.
  
  If required to give the performance of a *regression* model in a percentage, your best option would be the Mean Absolute Percentage Error ([MAPE](https://en.wikipedia.org/wiki/Mean_absolute_percentage_error)).
  
  
  ---
  
  