Answers.Body: <p>Different metrics measure different quantities, so there is no reason to expect different metrics to move together unless one is a function of the other (such as MSE and RMSE).</p>
  <p>Further, metrics like accuracy use thresholds that can prove problematic, versus evaluating the probability outputs directly. This is discussed extensively on the statistics Stack, Cross Validated. I will leave a few links.</p>
  <p><a href="https://stats.stackexchange.com/a/312787/247274">https://stats.stackexchange.com/a/312787/247274</a></p>
  <p><a href="https://stats.stackexchange.com/a/359936/247274">https://stats.stackexchange.com/a/359936/247274</a></p>
  <p><a href="https://stats.stackexchange.com/a/312124/247274">https://stats.stackexchange.com/a/312124/247274</a></p>
  <p>However, even strictly proper scoring rules do not have to move together. Since, for instance, Brier score and log loss (both of which are examples of the strictly proper scoring rules discussed in the first link) are measures of different quantities, this is desirable. Depending on what we value, we might prefer one over the other.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-23T22:48:28.597Z
Answers.LastActivityDate: 2022-09-24T15:06:44.870Z
Answers.LastEditDate: 2022-09-24T15:06:44.870Z
Answers.UserDisplayName: Dave
Body: <p>Recently I developed a custom training algorithm for deep learning models, based on evolutionary algorithms. Details are not important, except that it also uses decreasing regular cross entropy loss as its fitness function.</p>
  <p>What I observed is that it very well decreases the loss function but the classification metrics such as accuracy, precision or recall also decrease along the training. This got me confused, as I was sure that decreasing loss such as cross entropy should always increase these metrics. After researching I found out this is possible due to fact that cross entropy can decrease in case where confidence on few samples is greatly increasing, but many other samples are meanwhile getting incorrect scores, but the profit from these few correct ones are dominant over many incorrect ones: <a href="https://www.quora.com/What-is-the-matter-when-loss-decreases-and-accuracy-decreases-too-on-training-neural-network?top_ans=238980470" rel="nofollow noreferrer">https://www.quora.com/What-is-the-matter-when-loss-decreases-and-accuracy-decreases-too-on-training-neural-network?top_ans=238980470</a></p>
  <p>So my question is: is there a loss function that when decreasing will always be increasing classification metrics?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-24T23:59:41.670Z
Comments.Text: Does not answer the question (which I think is actually ill-posed, for reasons I hint at at the end of this answer of mine at SO), but may be helpful: https://stackoverflow.com/questions/47817424/loss-accuracy-are-these-reasonable-learning-curves/47819022#47819022
Comments.UserDisplayName: desertnaut
Tags: <deep-learning><training><loss><metric><cross-entropy>
Title: What loss function will be correlated with classification metrics?
UserDisplayName: GKozinski
ThreadText: # What loss function will be correlated with classification metrics?
  
  (Asked by: username_0 on 2022-09-23)
  
  Recently I developed a custom training algorithm for deep learning models, based on evolutionary algorithms. Details are not important, except that it also uses decreasing regular cross entropy loss as its fitness function.
  
  What I observed is that it very well decreases the loss function but the classification metrics such as accuracy, precision or recall also decrease along the training. This got me confused, as I was sure that decreasing loss such as cross entropy should always increase these metrics. After researching I found out this is possible due to fact that cross entropy can decrease in case where confidence on few samples is greatly increasing, but many other samples are meanwhile getting incorrect scores, but the profit from these few correct ones are dominant over many incorrect ones: <https://www.quora.com/What-is-the-matter-when-loss-decreases-and-accuracy-decreases-too-on-training-neural-network?top_ans=238980470>
  
  So my question is: is there a loss function that when decreasing will always be increasing classification metrics?
  
  **Tags:** deep-learning, training, loss, metric, cross-entropy
  > **Comments:**
  > **username_1**: Does not answer the question (which I think is actually ill-posed, for reasons I hint at at the end of this answer of mine at SO), but may be helpful: https://stackoverflow.com/questions/47817424/loss-accuracy-are-these-reasonable-learning-curves/47819022#47819022  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-09-23. Score: 2
  
  Different metrics measure different quantities, so there is no reason to expect different metrics to move together unless one is a function of the other (such as MSE and RMSE).
  
  Further, metrics like accuracy use thresholds that can prove problematic, versus evaluating the probability outputs directly. This is discussed extensively on the statistics Stack, Cross Validated. I will leave a few links.
  
  <https://stats.stackexchange.com/a/312787/247274>
  
  <https://stats.stackexchange.com/a/359936/247274>
  
  <https://stats.stackexchange.com/a/312124/247274>
  
  However, even strictly proper scoring rules do not have to move together. Since, for instance, Brier score and log loss (both of which are examples of the strictly proper scoring rules discussed in the first link) are measures of different quantities, this is desirable. Depending on what we value, we might prefer one over the other.
  
  
  ---
  
  