Answers.Body: <p>The test set should represents the <em>&quot;real&quot;</em> data distribution your model will tackle once deployed and used in real applications. So the quick answer is <strong>yes, the test data should be imbalanced</strong>, which comes also as a sort of forced choice for you considering the super small size of you dataset.</p>
  <p><strong>You want to keep in mind though that this makes a bit trickier to interpret the resulting metrics.</strong></p>
  <p>For example, if your test set is made of 90 instances labeled A and 10 instances labeled B, a model might predict class A for all instances and still have a final accuracy of 90%, correct, but completely unrepresentative of the real behavior of the model. So make sure to compute metrics like precision and recall for each class.</p>
  <p>Extra: considering the size of your dataset, you might consider using <a href="https://machinelearningmastery.com/k-fold-cross-validation/" rel="nofollow noreferrer">k-fold cross validation</a>. Split your dataset into k train-test subsets, train k models and average the resulting metrics. The logic being that with such few instances the dataset is almost guaranteed to contain biases of some sort.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-10T11:48:50.287Z
Answers.Text: If the real distribution is imbalanced, the test dataset should be imbalanced. However, note that, even if the training dataset is imbalanced (which is the case of the OP), this does not mean that the real distribution is imbalanced. So, if the real distribution isn't imbalanced, then one may want to consider oversampling and undersampling.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-09T08:44:43.730Z
Answers.LastActivityDate: 2021-11-09T08:55:28.240Z
Answers.LastEditDate: 2021-11-09T08:55:28.240Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I have a dataset that contains 560 datapoints, and I would like to do binary classification on it. 400 datapoints belong to class 1, and 160 points belong to class 2. In the case of an imbalanced dataset like this, how to arrange the test dataset to get valid performance results? Should I keep the same imbalanced data distribution for the test set which is similar to the distribution of the training data, or arrange it in a way that half of the test points belong to the first class and the remaining half belongs to the second class?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-10T12:02:48.033Z
Comments.Text: We had similar questions in the past. For example, see [here](https://ai.stackexchange.com/q/27404/2444).
Comments.UserDisplayName: nbro
Tags: <machine-learning><data-preprocessing><binary-classification><cross-validation><imbalanced-datasets>
Title: How to arrange test dataset distribution for an imbalanced classification problem?
UserDisplayName: Clara
ThreadText: # How to arrange test dataset distribution for an imbalanced classification problem?
  
  (Asked by: username_0 on 2021-11-08)
  
  I have a dataset that contains 560 datapoints, and I would like to do binary classification on it. 400 datapoints belong to class 1, and 160 points belong to class 2. In the case of an imbalanced dataset like this, how to arrange the test dataset to get valid performance results? Should I keep the same imbalanced data distribution for the test set which is similar to the distribution of the training data, or arrange it in a way that half of the test points belong to the first class and the remaining half belongs to the second class?
  
  **Tags:** machine-learning, data-preprocessing, binary-classification, cross-validation, imbalanced-datasets
  > **Comments:**
  > **username_1**: We had similar questions in the past. For example, see [here](https://ai.stackexchange.com/q/27404/2444).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-11-09. Score: 1 | Accepted answer
  
  The test set should represents the *"real"* data distribution your model will tackle once deployed and used in real applications. So the quick answer is **yes, the test data should be imbalanced**, which comes also as a sort of forced choice for you considering the super small size of you dataset.
  
  **You want to keep in mind though that this makes a bit trickier to interpret the resulting metrics.**
  
  For example, if your test set is made of 90 instances labeled A and 10 instances labeled B, a model might predict class A for all instances and still have a final accuracy of 90%, correct, but completely unrepresentative of the real behavior of the model. So make sure to compute metrics like precision and recall for each class.
  
  Extra: considering the size of your dataset, you might consider using [k-fold cross validation](https://machinelearningmastery.com/k-fold-cross-validation/). Split your dataset into k train-test subsets, train k models and average the resulting metrics. The logic being that with such few instances the dataset is almost guaranteed to contain biases of some sort.
  
  > **Comments:**
  > **username_1**: If the real distribution is imbalanced, the test dataset should be imbalanced. However, note that, even if the training dataset is imbalanced (which is the case of the OP), this does not mean that the real distribution is imbalanced. So, if the real distribution isn't imbalanced, then one may want to consider oversampling and undersampling.  
  
  ---
  
  