Answers.Body: <p>Okay, I think it's better if we distinguish loss and accuracy first via <a href="https://datascience.stackexchange.com/questions/42599/what-is-the-relationship-between-the-accuracy-and-the-loss-in-deep-learning#:%7E:text=There%20is%20no%20relationship%20between,you%20made%20on%20the%20data.">Jeremy's answer</a>, and I agree with him with the sentence <strong>&quot;low or huge loss is a subjective metric&quot;</strong>.</p>
  <p>The loss value is easy to affect by noise from data and significant increase with a few error data points. My advice in this case is to use more evaluation metrics, and understand correctly what you need from your model.</p>
  <p>For example, with Cifar 10, and you need the more correct label the better, you can believe in accuracy. However, if you want your model to make sure its result is the correct, area under receiver operating characteristic curve (AUROC) maybe the better choice.</p>
  <p>For example, classification problem with 3 class, correct label y = 1:</p>
  <ul>
  <li>Good accuracy, bad AUROC: the output probability from softmax [0.3,0.4,0.3]</li>
  <li>Good accuracy, good AUROC: [0.1,0.8,0.1]</li>
  </ul>
  <p>And with imbalanced dataset, Precision, Recall and F1-score will be more suitable.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-23T05:56:06.053Z
Answers.LastActivityDate: 2021-03-23T05:56:06.053Z
Answers.UserDisplayName: CuCaRot
Answers.Body: <p>In highly imbalanced classification problems, the highest accuracy can often be achieved simply by assigning the majority class to all observations. This is why learning algorithms do not maximize classification accuracy but minimize a loss function. 
  Fundamentally, loss functions capture how much you "lose" when there is a difference between the statistic you want to estimate and the estimate itself. The appropriate loss function is not given by nature, but is provided by you.</p>
  
  <p>With this in mind, it is not quite accurate to say that the highest accuracy is reached after 120 epochs: it is merely the maximum accuracy achieved by the algorithm so far. Unless you run the algorithm for longer, there is no way to say if this is even a local maximum. For example, assigning every observation to the majority class may well achieve a higher accuracy than that achieved at 120 epochs. The only significance of the 120 epochs is therefore that that is how long you ran the algorithm for.</p>
  
  <p>Given these considerations, it makes far more sense to stop at around 50 epochs, when your loss function is minimized.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-12T17:15:47.427Z
Answers.Text: Hi, you are right regarding the points for imbalanced datasets but Cifar10 isn't quite imbalanced. More or less, there are same amount of samples for each class.
Answers.UserDisplayName: SpiderRico
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-12T17:43:24.773Z
Answers.Text: It doesn't matter. What matters is that learning involves minimizing an appropriately chosen loss function. Unless you choose the loss function to be 1 - accuracy, minimizing losses does not correspond to maximizing accuracy.
Answers.UserDisplayName: perenniallydisappointed
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-12T17:51:13.380Z
Answers.Text: but your model tries to minimize the training loss, not the validation loss?
Answers.UserDisplayName: SpiderRico
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-13T08:55:54.333Z
Answers.Text: That's right. You want to minimize loss with respect to the data generating process, not with respect to the data itself. In other words: the model shouldn't just fit the data you already have, but also the data that you will get in the future. You can assess how well it does this on a test set.
Answers.UserDisplayName: perenniallydisappointed
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-25T19:23:22.730Z
Answers.Text: What about a better metric like F1? Would you ever prefer it over validation loss to save the best model?
Answers.UserDisplayName: rjurney
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-24T09:58:02.207Z
Answers.Text: This answer is not true. We optimize for a loss as the accuracy is not differentiable, thus one designs a surrogate loss function capturing accuracy as good as possible while staying differentiable. Most loss functions are optimal if one achieves maximal accuracy, i.e. do not help for imbalanced data. If one has imbalanced data, one has to adapt the loss (or employ other methods) to avoid learning a majority classifier.
Answers.UserDisplayName: saper0
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-12T16:42:42.247Z
Answers.LastActivityDate: 2020-02-12T16:42:42.247Z
Answers.UserDisplayName: perenniallydisappointed
Body: <p>I trained a ResNet20 on Cifar10 and obtained the following learning curves.</p>
  
  <p><a href="https://i.sstatic.net/O7JVh.png" rel="noreferrer"><img src="https://i.sstatic.net/O7JVh.png" alt="enter image description here"></a></p>
  
  <p>From the figures, I see at epoch 52, my validation loss is 0.323 (the lowest), and my validation accuracy is 89.7%.</p>
  
  <p>On the other hand, at the end of the training (epoch 120), my validation loss is 0.413 and my validation accuracy is 91.3% (the highest).</p>
  
  <p>Say I'd like to deploy this model on some real-world application. Should I prefer the snapshotted model at epoch 52, the one with lowest validation loss, or the model obtained at the end of training, the one with highest validation accuracy?</p>
  
Tags: <neural-networks><deep-learning><applications><overfitting><early-stopping>
Title: Should I prefer the model with the lowest validation loss or the highest validation accuracy to deploy?
UserDisplayName: SpiderRico
ThreadText: # Should I prefer the model with the lowest validation loss or the highest validation accuracy to deploy?
  
  (Asked by: username_0 on 2020-02-12)
  
  I trained a ResNet20 on Cifar10 and obtained the following learning curves.
  
  [![enter image description here](https://i.sstatic.net/O7JVh.png)](https://i.sstatic.net/O7JVh.png)
  
  From the figures, I see at epoch 52, my validation loss is 0.323 (the lowest), and my validation accuracy is 89.7%.
  
  On the other hand, at the end of the training (epoch 120), my validation loss is 0.413 and my validation accuracy is 91.3% (the highest).
  
  Say I'd like to deploy this model on some real-world application. Should I prefer the snapshotted model at epoch 52, the one with lowest validation loss, or the model obtained at the end of training, the one with highest validation accuracy?
  
  **Tags:** neural-networks, deep-learning, applications, overfitting, early-stopping
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-03-23. Score: 1 | Accepted answer
  
  Okay, I think it's better if we distinguish loss and accuracy first via [Jeremy's answer](https://datascience.stackexchange.com/questions/42599/what-is-the-relationship-between-the-accuracy-and-the-loss-in-deep-learning#:%7E:text=There%20is%20no%20relationship%20between,you%20made%20on%20the%20data.), and I agree with him with the sentence **"low or huge loss is a subjective metric"**.
  
  The loss value is easy to affect by noise from data and significant increase with a few error data points. My advice in this case is to use more evaluation metrics, and understand correctly what you need from your model.
  
  For example, with Cifar 10, and you need the more correct label the better, you can believe in accuracy. However, if you want your model to make sure its result is the correct, area under receiver operating characteristic curve (AUROC) maybe the better choice.
  
  For example, classification problem with 3 class, correct label y = 1:
  
  * Good accuracy, bad AUROC: the output probability from softmax [0.3,0.4,0.3]
  * Good accuracy, good AUROC: [0.1,0.8,0.1]
  
  And with imbalanced dataset, Precision, Recall and F1-score will be more suitable.
  
  
  ---
  
  ### Answer by username_2 on 2020-02-12. Score: 1
  
  In highly imbalanced classification problems, the highest accuracy can often be achieved simply by assigning the majority class to all observations. This is why learning algorithms do not maximize classification accuracy but minimize a loss function.
  Fundamentally, loss functions capture how much you "lose" when there is a difference between the statistic you want to estimate and the estimate itself. The appropriate loss function is not given by nature, but is provided by you.
  
  With this in mind, it is not quite accurate to say that the highest accuracy is reached after 120 epochs: it is merely the maximum accuracy achieved by the algorithm so far. Unless you run the algorithm for longer, there is no way to say if this is even a local maximum. For example, assigning every observation to the majority class may well achieve a higher accuracy than that achieved at 120 epochs. The only significance of the 120 epochs is therefore that that is how long you ran the algorithm for.
  
  Given these considerations, it makes far more sense to stop at around 50 epochs, when your loss function is minimized.
  
  > **Comments:**
  > **username_0**: Hi, you are right regarding the points for imbalanced datasets but Cifar10 isn't quite imbalanced. More or less, there are same amount of samples for each class.  
  > **username_2**: It doesn't matter. What matters is that learning involves minimizing an appropriately chosen loss function. Unless you choose the loss function to be 1 - accuracy, minimizing losses does not correspond to maximizing accuracy.  
  > **username_0**: but your model tries to minimize the training loss, not the validation loss?  
  > **username_2**: That's right. You want to minimize loss with respect to the data generating process, not with respect to the data itself. In other words: the model shouldn't just fit the data you already have, but also the data that you will get in the future. You can assess how well it does this on a test set.  
  > **username_3**: What about a better metric like F1? Would you ever prefer it over validation loss to save the best model?  
  > **username_4**: This answer is not true. We optimize for a loss as the accuracy is not differentiable, thus one designs a surrogate loss function capturing accuracy as good as possible while staying differentiable. Most loss functions are optimal if one achieves maximal accuracy, i.e. do not help for imbalanced data. If one has imbalanced data, one has to adapt the loss (or employ other methods) to avoid learning a majority classifier.  
  
  ---
  
  