Answers.Body: <p>As you point out, your results in this case are biased. Using skewed target distributions even with a simple train-test split would lead to training your model with more of one class than the other.</p>
  <p>There are multiple ways to prevent this, depending on how imbalanced (skewed) is your dataset. If the imbalance is not severe (e.g. a 40:60 target distribution) you could use:</p>
  <ul>
  <li>Cross-validation</li>
  <li>Stratified Cross-validation</li>
  <li>A stratified train-test split</li>
  </ul>
  <p>If the dataset is severely imbalanced, then you might train balancing techniques like:</p>
  <ul>
  <li>Perform under- or over-sampling. By the amount of data that you have, it might be better for you to consider under-sampling.</li>
  <li>Generate synthetic data based on prior knowledge</li>
  <li>Create synthetic data using SMOTE</li>
  <li>Use Penalised models that penalise making classification mistakes on the minority class</li>
  </ul>
  <p>Depending on your goal and the severity of the imbalance you might consider simply focusing on either a high precision or high recall. It might be also worth considering changing your perspective (i.e. maybe this is not a classification problem, it could be more of a &quot;detection&quot; problem)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-14T07:11:50.100Z
Answers.Text: Thanks. If we use the SMOTE, do we need to apply it to both the **training** and **test data**? Or only train data (if so, why not on test data), and by detection problem, do you mean we focus on high recall; that is, it is more critical to identify malicious nodes? 
  I will be using this in a resource-sharing environment,  so I believe it is equally essential that honest nodes are not misclassified as malicious so that their resources can be shared with other nodes.
Answers.UserDisplayName: Zal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-14T12:30:06.013Z
Answers.Text: SMOTE can be seen as a pre-processing step. As such, you always do the pre-processing fit with the training data. Your test set should always be left untouched until the very end.  In that way you can measure the actual performance of the model. If you use the test set to balance the data, you will introduce bias. Regarding the detection problem, yes, that what I meant. If both are important, then, try to shoot for a high f1-score rather than balancing them
Answers.UserDisplayName: DCrown
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-14T21:48:54.457Z
Answers.Text: I need a little clarification on the SMOTE thing. Let's say I made the data split  (70:30), and we have X_train, y_train, and X_test, y_test data. So first, I apply SMOTE to train data (X_train, y_train), then apply SMOTE to test data. Would using the SMOTE technique on the train and test data sets separately also introduce bias? 
   Is resampling (SMOTE) the test data not required? as we need to ensure that our model learns both classes, and how are test data looks does not really matter.
Answers.UserDisplayName: Zal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-17T08:44:36.367Z
Answers.Text: What would be the point of balancing the test set? Remember the test set in any machine learning problem is suppose to represent "new/never before seen" data. You want your model to perform well in the real world with real data. If the data acquired by itself is not balanced then that's a different discussion. The test set is only used to evaluate the true performance of the model, is not used to train it and make it learn. SMOTE creates synthetic data that helps during the training process.
Answers.UserDisplayName: DCrown
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-13T10:33:23.483Z
Answers.LastActivityDate: 2022-10-13T10:33:23.483Z
Answers.UserDisplayName: DCrown
Body: <p>I have an imbalanced dataset on intrusion detection. I have (attack class) 3668045 samples and (benign class) 477 samples. I made a 70:30 Train test split. My problem is to predict whether the given node belongs to the attack class or the benign class. As a first step, I trained a decision tree model on the dataset without using any balancing technique. I obtained the following results for my model on the test set using the sklearn metrics.</p>
  <pre><code>Scores for  Decision Tree
  Accuracy: 0.9998991419799247
  True positive 1100391
  True Negative 55
  False Positive 86
  False Negative 25
  F2-score 0.9999661949775551
  Precision 0.9999218520696025
  Recall 0.9999772813190648
  F1-score 0.9999495659261946
  Log loss: 0.0034835750853569407
  Decision Tree : AUROC (ROC Curve) = 0.999
  Decision Tree : AUPR(Precision/Recall curve) = 1.000
  
   Classification Report
                 precision    recall  f1-score   support
  
             0       0.69      0.39      0.50       141
             1       1.00      1.00      1.00   1100416
  
      accuracy                           1.00   1100557
     macro avg       0.84      0.70      0.75   1100557
  weighted avg       1.00      1.00      1.00   1100557
  
  </code></pre>
  <p>Why am I getting high, almost perfect AUROC and AUPR scores, even though the precision and recall for my minority class are very low? What measures can I take to improve the results such that they are not biased and my model is generalizing well? How can I ensure that?</p>
  
Tags: <binary-classification><decision-trees><anomaly-detection><scikit-learn><imbalanced-datasets>
Title: How to interpret binary classification metrics on an imbalanced data set?
UserDisplayName: Zal
ThreadText: # How to interpret binary classification metrics on an imbalanced data set?
  
  (Asked by: username_0 on 2022-10-12)
  
  I have an imbalanced dataset on intrusion detection. I have (attack class) 3668045 samples and (benign class) 477 samples. I made a 70:30 Train test split. My problem is to predict whether the given node belongs to the attack class or the benign class. As a first step, I trained a decision tree model on the dataset without using any balancing technique. I obtained the following results for my model on the test set using the sklearn metrics.
  
  ```
  Scores for  Decision Tree
  Accuracy: 0.9998991419799247
  True positive 1100391
  True Negative 55
  False Positive 86
  False Negative 25
  F2-score 0.9999661949775551
  Precision 0.9999218520696025
  Recall 0.9999772813190648
  F1-score 0.9999495659261946
  Log loss: 0.0034835750853569407
  Decision Tree : AUROC (ROC Curve) = 0.999
  Decision Tree : AUPR(Precision/Recall curve) = 1.000
  
   Classification Report
                 precision    recall  f1-score   support
  
             0       0.69      0.39      0.50       141
             1       1.00      1.00      1.00   1100416
  
      accuracy                           1.00   1100557
     macro avg       0.84      0.70      0.75   1100557
  weighted avg       1.00      1.00      1.00   1100557
  
  ```
  
  Why am I getting high, almost perfect AUROC and AUPR scores, even though the precision and recall for my minority class are very low? What measures can I take to improve the results such that they are not biased and my model is generalizing well? How can I ensure that?
  
  **Tags:** binary-classification, decision-trees, anomaly-detection, scikit-learn, imbalanced-datasets
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-10-13. Score: 0 | Accepted answer
  
  As you point out, your results in this case are biased. Using skewed target distributions even with a simple train-test split would lead to training your model with more of one class than the other.
  
  There are multiple ways to prevent this, depending on how imbalanced (skewed) is your dataset. If the imbalance is not severe (e.g. a 40:60 target distribution) you could use:
  
  * Cross-validation
  * Stratified Cross-validation
  * A stratified train-test split
  
  If the dataset is severely imbalanced, then you might train balancing techniques like:
  
  * Perform under- or over-sampling. By the amount of data that you have, it might be better for you to consider under-sampling.
  * Generate synthetic data based on prior knowledge
  * Create synthetic data using SMOTE
  * Use Penalised models that penalise making classification mistakes on the minority class
  
  Depending on your goal and the severity of the imbalance you might consider simply focusing on either a high precision or high recall. It might be also worth considering changing your perspective (i.e. maybe this is not a classification problem, it could be more of a "detection" problem)
  
  > **Comments:**
  > **username_0**: Thanks. If we use the SMOTE, do we need to apply it to both the **training** and **test data**? Or only train data (if so, why not on test data), and by detection problem, do you mean we focus on high recall; that is, it is more critical to identify malicious nodes?  I will be using this in a resource-sharing environment,  so I believe it is equally essential that honest nodes are not misclassified as malicious so that their resources can be shared with other nodes.  
  > **username_1**: SMOTE can be seen as a pre-processing step. As such, you always do the pre-processing fit with the training data. Your test set should always be left untouched until the very end.  In that way you can measure the actual performance of the model. If you use the test set to balance the data, you will introduce bias. Regarding the detection problem, yes, that what I meant. If both are important, then, try to shoot for a high f1-score rather than balancing them  
  > **username_0**: I need a little clarification on the SMOTE thing. Let's say I made the data split  (70:30), and we have X_train, y_train, and X_test, y_test data. So first, I apply SMOTE to train data (X_train, y_train), then apply SMOTE to test data. Would using the SMOTE technique on the train and test data sets separately also introduce bias?   Is resampling (SMOTE) the test data not required? as we need to ensure that our model learns both classes, and how are test data looks does not really matter.  
  > **username_1**: What would be the point of balancing the test set? Remember the test set in any machine learning problem is suppose to represent "new/never before seen" data. You want your model to perform well in the real world with real data. If the data acquired by itself is not balanced then that's a different discussion. The test set is only used to evaluate the true performance of the model, is not used to train it and make it learn. SMOTE creates synthetic data that helps during the training process.  
  
  ---
  
  