Answers.Body: <p>4861/5111 is about 95.1%, so it looks like your classifier is probably predicting every patient as &quot;no stroke&quot; (i.e. it is not really doing anything useful).  The thing to do is to work out the costs of false-positive (predicting a stroke but it didn't happen) and false-negative (predicting they won't have a stroke but they did) errors.  Then factor those into your training process.  This can be done in two ways: Firstly by using a weighted training criterion so you can factor the costs in explicitly; or secondly by resampling the dataset to have a greater proportion of &quot;stroke&quot; patients.</p>
  <p>The presence of an imbalance is <strong>very</strong> rarely a justification in itself for resampling the data.  The real reason to do so is that tasks with an imbalance very often have unequal misclassification costs, for instance if you tell someone they are going to have a stroke when they won't, you will have scared them rather badly, and you may spend some more money on more testing, but that is probably about it.  If you tell someone they are not at risk of a stroke when they are, they may go home, have a stroke and become severely disabled as a result, or even die.  So the false-negative cost is likely to be much higher and that will boost recognition of the positive cases.  The amount of resampling required depends <strong>only</strong> on the costs; the degree of imbalance is entirely irrelevant.</p>
  <p>BTW, rather than look at accuracy, you might want to look at a related metric, which is the improvement over a classifier that always predicts the majority class.  Something like:</p>
  <p><span class="math-container">$$\frac{\mathrm{Accuracy} - \pi}{1 - \pi}$$</span></p>
  <p>where <span class="math-container">$\pi$</span> is the proportion of the most common class in the dataset.  In this case, your classifier is going to score somewhere close to zero as it is probably just going by the majority class, and a score of zero shows clearly that it isn't doing a good job.  A score of 1 would be perfect classification.  A negative score shows the model is worse than just guessing.  It is an affine transformation of accuracy though, so it is still measuring the same basic thing, just on a more interpretable scale.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-28T16:42:23.940Z
Answers.LastActivityDate: 2022-04-28T16:47:50.943Z
Answers.LastEditDate: 2022-04-28T16:47:50.943Z
Answers.UserDisplayName: Dikran Marsupial
Answers.Body: <p>You should use another classification metric for evaluating your model. I would just look at the <em>confusion matrix</em> to see how the model performance on the &quot;interesting class&quot; (minority class).</p>
  <p>To overcome your imbalanced dataset, you could upsample the minority class. Bootstrapping is a great staring point. Then advance with SMOTE or something else. This article might give you some ides; <a href="https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/" rel="nofollow noreferrer">8 Tactics to Combat Imbalanced Classes in Your Machine Learning Dataset</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-29T09:18:59.847Z
Answers.Text: The classification metric should depend on the needs of the application rather than problems with the model.  In this application the expected loss is likely to be a better metric as the misclassification costs are obviously unequal, so the thing to do is to work out what those costs reasonably might be.
Answers.UserDisplayName: Dikran Marsupial
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-29T08:52:18.363Z
Answers.LastActivityDate: 2022-04-29T08:52:18.363Z
Answers.UserDisplayName: Adrian Evensen
Answers.Body: <p>You can use a data augmentation technique like SMOTE to oversample the minority class. It will help you have a more balanced dataset. Here is a nice guide on it:
  <a href="https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/" rel="nofollow noreferrer">https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-29T09:21:06.977Z
Answers.Text: I'd advise against SMOTE for modern classifier systems.  It was developed for very simple classifiers that didn't support cost-sensitive learning or have some means of avoiding over-fitting.  Modern classifier systems, such as SVMs or NN have both, so SMOTE is not likely to be necessary (or helpful as there is a lot of good theory supporting regularisation, but none for the rather peculiar method SMOTE uses to generate synthetic examples)
Answers.UserDisplayName: Dikran Marsupial
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-29T04:20:42.520Z
Answers.LastActivityDate: 2022-04-29T04:20:42.520Z
Answers.UserDisplayName: rachkov91
Body: <p>I'm constructing a feed forward neural network that predicts whether a patient will get a stroke or not. However, my dataset is very unbalanced. Out of 5111 rows, 250 contain patients that have had a stroke (1) and 4861 that did not (0). The accuracy is (as a result of this, I suppose) very high (89% on the first epoch, and 95% on the second, then it stays at 95%). What would be the best thing to do about this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-28T14:26:59.273Z
Comments.Text: Please, put your **specific question** in the title. "Unbalanced dataset" is not specific and not a question. Moreover, use more appropriate tags, like [tag:unbalanced-datasets]. Finally, I recommend that you take a look at existing questions. I think this question was already asked more than once here in the past.
Comments.UserDisplayName: nbro
Tags: <neural-networks><machine-learning><datasets><feedforward-neural-networks><imbalanced-datasets>
Title: How to deal with an unbalanced dataset?
UserDisplayName: JanHudec
ThreadText: # How to deal with an unbalanced dataset?
  
  (Asked by: username_0 on 2022-04-28)
  
  I'm constructing a feed forward neural network that predicts whether a patient will get a stroke or not. However, my dataset is very unbalanced. Out of 5111 rows, 250 contain patients that have had a stroke (1) and 4861 that did not (0). The accuracy is (as a result of this, I suppose) very high (89% on the first epoch, and 95% on the second, then it stays at 95%). What would be the best thing to do about this?
  
  **Tags:** neural-networks, machine-learning, datasets, feedforward-neural-networks, imbalanced-datasets
  
  > **Comments:**
  > **username_1**: Please, put your **specific question** in the title. "Unbalanced dataset" is not specific and not a question. Moreover, use more appropriate tags, like [tag:unbalanced-datasets]. Finally, I recommend that you take a look at existing questions. I think this question was already asked more than once here in the past.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2022-04-28. Score: 1
  
  4861/5111 is about 95.1%, so it looks like your classifier is probably predicting every patient as "no stroke" (i.e. it is not really doing anything useful). The thing to do is to work out the costs of false-positive (predicting a stroke but it didn't happen) and false-negative (predicting they won't have a stroke but they did) errors. Then factor those into your training process. This can be done in two ways: Firstly by using a weighted training criterion so you can factor the costs in explicitly; or secondly by resampling the dataset to have a greater proportion of "stroke" patients.
  
  The presence of an imbalance is **very** rarely a justification in itself for resampling the data. The real reason to do so is that tasks with an imbalance very often have unequal misclassification costs, for instance if you tell someone they are going to have a stroke when they won't, you will have scared them rather badly, and you may spend some more money on more testing, but that is probably about it. If you tell someone they are not at risk of a stroke when they are, they may go home, have a stroke and become severely disabled as a result, or even die. So the false-negative cost is likely to be much higher and that will boost recognition of the positive cases. The amount of resampling required depends **only** on the costs; the degree of imbalance is entirely irrelevant.
  
  BTW, rather than look at accuracy, you might want to look at a related metric, which is the improvement over a classifier that always predicts the majority class. Something like:
  
  $$\frac{\mathrm{Accuracy} - \pi}{1 - \pi}$$
  
  where $\pi$ is the proportion of the most common class in the dataset. In this case, your classifier is going to score somewhere close to zero as it is probably just going by the majority class, and a score of zero shows clearly that it isn't doing a good job. A score of 1 would be perfect classification. A negative score shows the model is worse than just guessing. It is an affine transformation of accuracy though, so it is still measuring the same basic thing, just on a more interpretable scale.
  
  
  ---
  
  ### Answer by username_3 on 2022-04-29. Score: 0
  
  You should use another classification metric for evaluating your model. I would just look at the *confusion matrix* to see how the model performance on the "interesting class" (minority class).
  
  To overcome your imbalanced dataset, you could upsample the minority class. Bootstrapping is a great staring point. Then advance with SMOTE or something else. This article might give you some ides; [8 Tactics to Combat Imbalanced Classes in Your Machine Learning Dataset](https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/)
  
  > **Comments:**
  > **username_2**: The classification metric should depend on the needs of the application rather than problems with the model.  In this application the expected loss is likely to be a better metric as the misclassification costs are obviously unequal, so the thing to do is to work out what those costs reasonably might be.  
  
  ---
  
  ### Answer by username_4 on 2022-04-29. Score: 0
  
  You can use a data augmentation technique like SMOTE to oversample the minority class. It will help you have a more balanced dataset. Here is a nice guide on it:
  <https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/>
  
  > **Comments:**
  > **username_2**: I'd advise against SMOTE for modern classifier systems.  It was developed for very simple classifiers that didn't support cost-sensitive learning or have some means of avoiding over-fitting.  Modern classifier systems, such as SVMs or NN have both, so SMOTE is not likely to be necessary (or helpful as there is a lot of good theory supporting regularisation, but none for the rather peculiar method SMOTE uses to generate synthetic examples)  
  
  ---
  
  