Answers.Body: <p>To me your evaluation looks good. And I wouldn't say that all these metrics are not good for imbalanced problems. Especially F1 score and ROC are actually pretty good indicators for spotting biases in a model.</p>
  <p>So unless you did a silly mistake like evaluating on training data you can say you trained a very good model and the metrics prove it. Now the question is if this model will generalize well also in a real use case scenario. I see that you have only 141 instances for the minority class in your test set. You don't say what type of data you're analyzing so we can't tell if the variability within that minority class is expected to be large or not. If you have reason to think it is, then you'll probably experience a drop in performance when using the model for real. Nothing in contrast with you're current evaluation, simply the results of testing on more representative data than your current test set.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-04T08:34:49.087Z
Answers.LastActivityDate: 2022-11-04T08:34:49.087Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I have trained a Decision Tree model on an <strong>imbalanced</strong> dataset. I got the following results for the test set from the sklearn and imblearn classification reports (attached below). Moreover, the other scores I calculated from sklearn  are as follows:</p>
  <p>AUROC (Area under Reciever operating curve) = 0.979</p>
  <p>AUPR (Area under Precision recall curve) = 1.000</p>
  <p>Recall = 0.99</p>
  <p>Precision = 0.99</p>
  <p>F1-score = 0.99</p>
  <p>F2-score = 0.99</p>
  <p>In my case, <strong>both classes are equally important</strong>. I have read that micro metrics (Prec, Recall, F1, F2, AUC) may not be good performance measures in imbalanced datasets as they can favor the majority class. For <strong>macro metrics</strong>, it is said that these are suited for <strong>multiclassification</strong> problems and may give a <strong>false narrative</strong> of the classifier in case we use them in binary classification. I am really confused if my classifier performing well or not. If it is performing <strong>well</strong> on which <strong>metrics</strong> should I justify this? Or how can I ensure my model is not <strong>biased?</strong></p>
  <pre><code>Classification Report
                 precision    recall  f1-score   support
  
             0       0.98      0.96      0.97       141
             1       1.00      1.00      1.00   1100416
  
      accuracy                           1.00   1100557
     macro avg       0.99      0.98      0.98   1100557
  weighted avg       1.00      1.00      1.00   1100557
  
  **************************************************************************************************
  
   Classification report imbalanced
                      pre       rec       spe        f1       geo       iba       sup
  
            0       0.98      0.96      1.00      0.97      0.98      0.95       141
            1       1.00      1.00      0.96      1.00      0.98      0.96   1100416
  
  avg / total       1.00      1.00      0.96      1.00      0.98      0.96   1100557
  
  <span class="math-container">```</span>
  </code></pre>
  
Tags: <binary-classification><decision-trees><imbalanced-datasets><scikit-learn><bias-variance-tradeoff>
Title: How to evaluate binary classifier on imbalanced dataset?
UserDisplayName: Zal
ThreadText: # How to evaluate binary classifier on imbalanced dataset?
  
  (Asked by: username_0 on 2022-11-04)
  
  I have trained a Decision Tree model on an **imbalanced** dataset. I got the following results for the test set from the sklearn and imblearn classification reports (attached below). Moreover, the other scores I calculated from sklearn are as follows:
  
  AUROC (Area under Reciever operating curve) = 0.979
  
  AUPR (Area under Precision recall curve) = 1.000
  
  Recall = 0.99
  
  Precision = 0.99
  
  F1-score = 0.99
  
  F2-score = 0.99
  
  In my case, **both classes are equally important**. I have read that micro metrics (Prec, Recall, F1, F2, AUC) may not be good performance measures in imbalanced datasets as they can favor the majority class. For **macro metrics**, it is said that these are suited for **multiclassification** problems and may give a **false narrative** of the classifier in case we use them in binary classification. I am really confused if my classifier performing well or not. If it is performing **well** on which **metrics** should I justify this? Or how can I ensure my model is not **biased?**
  
  ```
  Classification Report
                 precision    recall  f1-score   support
  
             0       0.98      0.96      0.97       141
             1       1.00      1.00      1.00   1100416
  
      accuracy                           1.00   1100557
     macro avg       0.99      0.98      0.98   1100557
  weighted avg       1.00      1.00      1.00   1100557
  
  **************************************************************************************************
  
   Classification report imbalanced
                      pre       rec       spe        f1       geo       iba       sup
  
            0       0.98      0.96      1.00      0.97      0.98      0.95       141
            1       1.00      1.00      0.96      1.00      0.98      0.96   1100416
  
  avg / total       1.00      1.00      0.96      1.00      0.98      0.96   1100557
  
  ```
  
  ```
  
  **Tags:** binary-classification, decision-trees, imbalanced-datasets, scikit-learn, bias-variance-tradeoff
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-11-04. Score: 0 | Accepted answer
  
  To me your evaluation looks good. And I wouldn't say that all these metrics are not good for imbalanced problems. Especially F1 score and ROC are actually pretty good indicators for spotting biases in a model.
  
  So unless you did a silly mistake like evaluating on training data you can say you trained a very good model and the metrics prove it. Now the question is if this model will generalize well also in a real use case scenario. I see that you have only 141 instances for the minority class in your test set. You don't say what type of data you're analyzing so we can't tell if the variability within that minority class is expected to be large or not. If you have reason to think it is, then you'll probably experience a drop in performance when using the model for real. Nothing in contrast with you're current evaluation, simply the results of testing on more representative data than your current test set.
  
  
  ---
  
  