Answers.Body: <p>You should choose the model A. The loss is just a differentiable proxy for accuracy.</p>
  
  <p>That said, the situation should be examined in more detail. If the higher loss is due to the data term, examine the data which produce high loss and check for presence of overfitting or incorrect labels. </p>
  
  <p>If the higher loss is due to a regularizer then reducing the regularization factor may further improve the results.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-23T17:51:15.900Z
Answers.Text: What do you mean precisely by "X is a _differentiable_ proxy for Y"? I think you should clarify this.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-23T20:26:56.453Z
Answers.Text: It means that a differentiable function X (eg. the negative log loss) is used as a replacement (proxy) for Y (accuracy) which is unsuitable due to being non-differentiable.
Answers.UserDisplayName: ssegvic
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-23T20:32:44.910Z
Answers.Text: But if there is a "one-to-one" correspondence between accuracy and loss, then why can you obtain the results the OP is describing?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-23T21:08:01.013Z
Answers.Text: A proxy is... ...just a proxy (an upper bound to be more precise). Not the real thing. A single datapoint may result in a very high loss contribution while only mildly affecting accuracy. One also may have high loss due to regularization term even when the accuracy is 100%. The only way to find out the reason for the discrepancy is to have a look at the data.
Answers.UserDisplayName: ssegvic
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-15T10:43:21.157Z
Answers.LastActivityDate: 2019-02-15T18:17:09.667Z
Answers.LastEditDate: 2019-02-15T18:17:09.667Z
Answers.UserDisplayName: ssegvic
Answers.Body: <p>It depends on your application! Imagine a binary classifier that is always very "confident" - it always assigns P=100% to Class A and 0% to Class B, or <em>vice versa</em> (sometimes wrong, never uncertain!).  Now imagine a "humble" model that is perhaps fractionally less accurate, but whose probabilities are actually meaningful (when it says "Class A with probability 70%" it is wrong 30% of the time).</p>
  
  <p>In your case, both losses are quite small, so we probably prefer the more accurate one.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-15T11:07:42.003Z
Answers.LastActivityDate: 2019-02-15T11:07:42.003Z
Answers.UserDisplayName: Edward Dixon
Answers.Body: <p>You should note that both your results are consistent with a "true" probability of 87% accuracy, and your measurement of a difference between these models is not statistically significant. With an 87% accuracy applied at random, then there is approx 14% chance of getting the two extremes of accuracy you have observed by chance if samples are chosen randomly from the target population, and models are different enough make errors effectively at random. This last assertion is usually not true though, so you can relax a little - that is, unless you took different random slices for cross-validation in each case.</p>
  
  <p>100 test cases is not really enough to discern small differences between models. I would suggest using k-fold cross-validation in order to reduce errors in your accuracy and loss estimates.</p>
  
  <p>Also, it is critical to check that the cross-validation split was identical in both cases here. If you have used auto-splitting with a standard tool and not set the appropriate RNG seed, then you may have got a different set each time, and your results are just showing you variance due to the validation split which could completely swamp any differences between the models.</p>
  
  <p>However, assuming the exact same dataset was used each time, and it was representative sample of your target population, then on average you should expect the one with the best metric to have the highest chance of <em>being</em> the best model.</p>
  
  <p>What you should really do is decide which metric to base the choice on <em>in advance of the experiment</em>. The metric should match some business goal for the model.</p>
  
  <p>Now you are trying to choose after the fact, you should go back to the reason you created the model in the first place and see if you can identify the correct metric. It might not be either accuracy or loss.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-18T14:45:32.943Z
Answers.Text: thank you! it's a nice Idea to use K-Fold CV. I use my model for industrial case, it's a text classification case. Some people said choose the smallest loss because high accuracy can lead to overfitting in the future, is that always true?
Answers.UserDisplayName: malioboro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-18T14:54:00.790Z
Answers.Text: @malioboro: I don't think that is true. Without any theory to back it up, it is an empty statement in any case. I could equally say "choose the highest accuracy, because low loss can lead to overfitting in future". What do your "some people" say about that - i.e. what is their theory, and under what assumptions does it work? Either way, ignore them and choose the correct metric for your problem as suggested by its intended use.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-20T08:53:24.560Z
Answers.Text: @NeilSlater Ah.. I see.. so there aren't any theory that said lower loss is safe from overfitting. Could you please give a few examples of problem and its correct metric in your answer? I'm not pretty sure with the metric for my problem
Answers.UserDisplayName: malioboro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-23T17:55:34.653Z
Answers.Text: Where does "You should note that both your results are consistent with a "true" probability of 87% accuracy, and your measurement of a difference between these models is not statistically significant." come from? Can you explain a little better these sentence in relation to the results presented by the OP?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-23T18:07:51.087Z
Answers.Text: @nbro: It comes from a very brief experiment assuming that the accuracy is 87%. This is probably not strictly valid mathematically, I simply ran a few 1000 trials with p =0.87 and 100 tests, and counted number of times it saw 91+ or 83-. That's more than a bit hacky, so I don't really want to go into details of that. The take-away is that error bars on the "91% accuracy" measurement using only 100 test samples are large.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-15T19:40:51.510Z
Answers.LastActivityDate: 2019-02-15T19:40:51.510Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have two Machine Learning models (I use LSTM) that have a different result on the validation set (~100 samples data):</p>
  <ul>
  <li>Model A: Accuracy: ~91%, Loss: ~0.01</li>
  <li>Model B: Accuracy: ~83%, Loss: ~0.003</li>
  </ul>
  <p>The size and the speed of both models are almost the same. So, which model should I choose?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-15T10:31:26.673Z
Comments.Text: The last question is not clear, please clarify.
Comments.UserDisplayName: ssegvic
Tags: <deep-learning><long-short-term-memory><loss><accuracy><validation>
Title: Should I choose a model with the smallest loss or highest accuracy?
UserDisplayName: malioboro
ThreadText: # Should I choose a model with the smallest loss or highest accuracy?
  
  (Asked by: username_0 on 2019-02-07)
  
  I have two Machine Learning models (I use LSTM) that have a different result on the validation set (~100 samples data):
  
  * Model A: Accuracy: ~91%, Loss: ~0.01
  * Model B: Accuracy: ~83%, Loss: ~0.003
  
  The size and the speed of both models are almost the same. So, which model should I choose?
  
  **Tags:** deep-learning, long-short-term-memory, loss, accuracy, validation
  
  > **Comments:**
  > **username_1**: The last question is not clear, please clarify.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2019-02-15. Score: 5 | Accepted answer
  
  You should note that both your results are consistent with a "true" probability of 87% accuracy, and your measurement of a difference between these models is not statistically significant. With an 87% accuracy applied at random, then there is approx 14% chance of getting the two extremes of accuracy you have observed by chance if samples are chosen randomly from the target population, and models are different enough make errors effectively at random. This last assertion is usually not true though, so you can relax a little - that is, unless you took different random slices for cross-validation in each case.
  
  100 test cases is not really enough to discern small differences between models. I would suggest using k-fold cross-validation in order to reduce errors in your accuracy and loss estimates.
  
  Also, it is critical to check that the cross-validation split was identical in both cases here. If you have used auto-splitting with a standard tool and not set the appropriate RNG seed, then you may have got a different set each time, and your results are just showing you variance due to the validation split which could completely swamp any differences between the models.
  
  However, assuming the exact same dataset was used each time, and it was representative sample of your target population, then on average you should expect the one with the best metric to have the highest chance of *being* the best model.
  
  What you should really do is decide which metric to base the choice on *in advance of the experiment*. The metric should match some business goal for the model.
  
  Now you are trying to choose after the fact, you should go back to the reason you created the model in the first place and see if you can identify the correct metric. It might not be either accuracy or loss.
  
  > **Comments:**
  > **username_0**: thank you! it's a nice Idea to use K-Fold CV. I use my model for industrial case, it's a text classification case. Some people said choose the smallest loss because high accuracy can lead to overfitting in the future, is that always true?  
  > **username_4**: @username_0: I don't think that is true. Without any theory to back it up, it is an empty statement in any case. I could equally say "choose the highest accuracy, because low loss can lead to overfitting in future". What do your "some people" say about that - i.e. what is their theory, and under what assumptions does it work? Either way, ignore them and choose the correct metric for your problem as suggested by its intended use.  
  > **username_0**: @username_4 Ah.. I see.. so there aren't any theory that said lower loss is safe from overfitting. Could you please give a few examples of problem and its correct metric in your answer? I'm not pretty sure with the metric for my problem  
  > **username_2**: Where does "You should note that both your results are consistent with a "true" probability of 87% accuracy, and your measurement of a difference between these models is not statistically significant." come from? Can you explain a little better these sentence in relation to the results presented by the OP?  
  > **username_4**: @username_2: It comes from a very brief experiment assuming that the accuracy is 87%. This is probably not strictly valid mathematically, I simply ran a few 1000 trials with p =0.87 and 100 tests, and counted number of times it saw 91+ or 83-. That's more than a bit hacky, so I don't really want to go into details of that. The take-away is that error bars on the "91% accuracy" measurement using only 100 test samples are large.  
  
  ---
  
  ### Answer by username_1 on 2019-02-15. Score: 6
  
  You should choose the model A. The loss is just a differentiable proxy for accuracy.
  
  That said, the situation should be examined in more detail. If the higher loss is due to the data term, examine the data which produce high loss and check for presence of overfitting or incorrect labels.
  
  If the higher loss is due to a regularizer then reducing the regularization factor may further improve the results.
  
  > **Comments:**
  > **username_2**: What do you mean precisely by "X is a _differentiable_ proxy for Y"? I think you should clarify this.  
  > **username_1**: It means that a differentiable function X (eg. the negative log loss) is used as a replacement (proxy) for Y (accuracy) which is unsuitable due to being non-differentiable.  
  > **username_2**: But if there is a "one-to-one" correspondence between accuracy and loss, then why can you obtain the results the OP is describing?  
  > **username_1**: A proxy is... ...just a proxy (an upper bound to be more precise). Not the real thing. A single datapoint may result in a very high loss contribution while only mildly affecting accuracy. One also may have high loss due to regularization term even when the accuracy is 100%. The only way to find out the reason for the discrepancy is to have a look at the data.  
  
  ---
  
  ### Answer by username_3 on 2019-02-15. Score: 1
  
  It depends on your application! Imagine a binary classifier that is always very "confident" - it always assigns P=100% to Class A and 0% to Class B, or *vice versa* (sometimes wrong, never uncertain!). Now imagine a "humble" model that is perhaps fractionally less accurate, but whose probabilities are actually meaningful (when it says "Class A with probability 70%" it is wrong 30% of the time).
  
  In your case, both losses are quite small, so we probably prefer the more accurate one.
  
  
  ---
  
  