Answers.Body: <p>If you are certain that your calculations are correct, then there may be class imbalance in your dataset.  The metrics <code>precision</code>, <code>f1_score</code>, and <code>accuracy</code> are dominated by the model's performance on the major class in your dataset.</p>
  <p>For example, if your dataset has <span class="math-container">$99$</span> examples of the positive class and <span class="math-container">$1$</span> example of the negative class and your model predicts positive <span class="math-container">$100$</span>% of the time (no matter what), then</p>
  <ol>
  <li><span class="math-container">$Precision = \frac{TP}{(TP + FP)} = \frac{99}{(99 + 1)} = \frac{99}{100}$</span></li>
  <li><span class="math-container">$F_1 score = 2\times\frac{TP}{(2 \times TP + FP + FN)} = 2 \times \frac{99}{(2 \times 99 + 1 + 0)} = 198/199$</span></li>
  <li><span class="math-container">$Accuracy = \frac{(TP + TN)}{(TP + TN + FP + FN)} = \frac{(99 + 0)}{(99 + 0 + 1 + 0)} = \frac{99}{100}$</span></li>
  </ol>
  <p>These give a false impression that your model is performing very well, but in fact, all it is doing is calling every example positive. This is why it is imperative to have a clear understanding of your performance metrics, what they mean, and what their limitations are.  In general, it is also a good idea to examine multiple performance metrics, if appropriate.</p>
  <p>In this example, specificity will reveal problems in the model and/or data.</p>
  <ol>
  <li><span class="math-container">$Specificity = \frac{TN}{(TN + FP)} = \frac{0}{(0 + 1)} = \frac{0}{1}$</span></li>
  </ol>
  <p>Given all possible positive-negative example pairs in your dataset, the <code>roc_auc</code> is the proportion of those pairs for which the model correctly identifies the positive example.  Thus, the AUC is more sensitive to class imbalance than those other metrics.</p>
  <p>Your results also indicates that your model is not doing any better than the flip of a coin because an AUC of <span class="math-container">$0.5$</span> indicates the expected performance of a dummy (naïve) classifier.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-13T15:38:15.923Z
Answers.LastActivityDate: 2022-11-13T17:20:36.177Z
Answers.LastEditDate: 2022-11-13T17:20:36.177Z
Answers.UserDisplayName: Snehal Patel
Body: <p><a href="https://i.sstatic.net/E8xTi.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/E8xTi.png" alt="enter image description here" /></a></p>
  <p>What does it mean when my precision and so on are so high and the roc auc score is around 0.5?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-13T15:27:46.103Z
Comments.Text: How many examples of each category do you have?
Comments.UserDisplayName: Dave
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-13T15:28:58.187Z
Comments.Text: do you mean sample size?
Comments.UserDisplayName: ProgrammingBot
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-13T15:29:29.963Z
Comments.Text: Broken down by category…I’m curious about how much imbalance you have in your data.
Comments.UserDisplayName: Dave
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-13T15:40:10.133Z
Comments.Text: i think i did something wrong. sorry for the disturbance
Comments.UserDisplayName: ProgrammingBot
Tags: <machine-learning><performance>
Title: What does that mean if my precision, F1-score are very high, but my ROC AUC score is around 0.5?
UserDisplayName: ProgrammingBot
ThreadText: # What does that mean if my precision, F1-score are very high, but my ROC AUC score is around 0.5?
  
  (Asked by: username_0 on 2022-11-13)
  
  [![enter image description here](https://i.sstatic.net/E8xTi.png)](https://i.sstatic.net/E8xTi.png)
  
  What does it mean when my precision and so on are so high and the roc auc score is around 0.5?
  
  **Tags:** machine-learning, performance
  
  > **Comments:**
  > **username_1**: How many examples of each category do you have?  
  > **username_0**: do you mean sample size?  
  > **username_1**: Broken down by category…I’m curious about how much imbalance you have in your data.  
  > **username_0**: i think i did something wrong. sorry for the disturbance  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-11-13. Score: 2
  
  If you are certain that your calculations are correct, then there may be class imbalance in your dataset. The metrics `precision`, `f1_score`, and `accuracy` are dominated by the model's performance on the major class in your dataset.
  
  For example, if your dataset has $99$ examples of the positive class and $1$ example of the negative class and your model predicts positive $100$% of the time (no matter what), then
  
  1. $Precision = \frac{TP}{(TP + FP)} = \frac{99}{(99 + 1)} = \frac{99}{100}$
  2. $F\_1 score = 2\times\frac{TP}{(2 \times TP + FP + FN)} = 2 \times \frac{99}{(2 \times 99 + 1 + 0)} = 198/199$
  3. $Accuracy = \frac{(TP + TN)}{(TP + TN + FP + FN)} = \frac{(99 + 0)}{(99 + 0 + 1 + 0)} = \frac{99}{100}$
  
  These give a false impression that your model is performing very well, but in fact, all it is doing is calling every example positive. This is why it is imperative to have a clear understanding of your performance metrics, what they mean, and what their limitations are. In general, it is also a good idea to examine multiple performance metrics, if appropriate.
  
  In this example, specificity will reveal problems in the model and/or data.
  
  1. $Specificity = \frac{TN}{(TN + FP)} = \frac{0}{(0 + 1)} = \frac{0}{1}$
  
  Given all possible positive-negative example pairs in your dataset, the `roc_auc` is the proportion of those pairs for which the model correctly identifies the positive example. Thus, the AUC is more sensitive to class imbalance than those other metrics.
  
  Your results also indicates that your model is not doing any better than the flip of a coin because an AUC of $0.5$ indicates the expected performance of a dummy (naïve) classifier.
  
  
  ---
  
  