Answers.Body: <p>It is straightforward to combine k-fold cross-validation with a technique like oversampling or undersampling. </p>
  
  <p>First, apply the balance-restoration technique to your training data. Then parametrize a model using k-fold cross-validation on the re-balanced training data. In Scikit learn, I believe you can even bundle these actions together into a single 'pipeline' object to make it easier to manipulate.</p>
  
  <p>Precision/recall is probably a fine starting place for measuring performance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-12T21:12:53.370Z
Answers.Text: So you mean first divide data into train and test set (hold out), rebalance train with Smote (for example) and then apply cross validation on this train set. What about instead, do oversample technique only during k fold iteration? let me explain better. Suppose i have unbalanced train, i apply 10-fold cross validation, so 9 groups will be used as train and 1 as test, what i mean is to apply oversample only at this time on the 9 train groups, just to reduce overfitting maybe. I hope i have been clear. i will wait your advices.  thanks
Answers.UserDisplayName: Alfonso
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-12T21:14:23.650Z
Answers.Text: @Alfonso You're quite right. That would be a better idea. My protocol leaks information from the validation set and yours does not.
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-11T18:13:08.523Z
Answers.LastActivityDate: 2019-03-11T18:13:08.523Z
Answers.UserDisplayName: John Doucette
Body: <p>I am dealing with an intent classification task on an Italian customer service data set.</p>
  <p>I've more or less 1.5k sentences and 29 classes (imbalanced).</p>
  <p>According to the literature, a good choice is to generate synthetic data, oversampling, or undersampling the training data, using for example the <a href="https://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis#SMOTE" rel="nofollow noreferrer">SMOTE algorithm</a>.</p>
  <p>I also want to use a cross-validation mechanism (<a href="https://en.wikipedia.org/wiki/Cross-validation_(statistics)#k-fold_cross-validation" rel="nofollow noreferrer">stratified <em>k</em>-fold</a>) to be more confident in the obtained result.</p>
  <p>I also know that accuracy is not the right metric to take into account, I should use precision, recall, and confusion matrix.</p>
  <p>Is it possible to combine k-fold cross-validation and oversampling (or undersampling) techniques?</p>
  
Tags: <imbalanced-datasets><multiclass-classification><k-fold-cv>
Title: Is it possible to combine k-fold cross-validation and oversampling for a multi-class text classification task with imbalanced data?
UserDisplayName: Alfonso
ThreadText: # Is it possible to combine k-fold cross-validation and oversampling for a multi-class text classification task with imbalanced data?
  
  (Asked by: username_0 on 2019-03-11)
  
  I am dealing with an intent classification task on an Italian customer service data set.
  
  I've more or less 1.5k sentences and 29 classes (imbalanced).
  
  According to the literature, a good choice is to generate synthetic data, oversampling, or undersampling the training data, using for example the [SMOTE algorithm](https://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis#SMOTE).
  
  I also want to use a cross-validation mechanism ([stratified *k*-fold](https://en.wikipedia.org/wiki/Cross-validation_(statistics)#k-fold_cross-validation)) to be more confident in the obtained result.
  
  I also know that accuracy is not the right metric to take into account, I should use precision, recall, and confusion matrix.
  
  Is it possible to combine k-fold cross-validation and oversampling (or undersampling) techniques?
  
  **Tags:** imbalanced-datasets, multiclass-classification, k-fold-cv
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-11. Score: 0 | Accepted answer
  
  It is straightforward to combine k-fold cross-validation with a technique like oversampling or undersampling.
  
  First, apply the balance-restoration technique to your training data. Then parametrize a model using k-fold cross-validation on the re-balanced training data. In Scikit learn, I believe you can even bundle these actions together into a single 'pipeline' object to make it easier to manipulate.
  
  Precision/recall is probably a fine starting place for measuring performance.
  
  > **Comments:**
  > **username_0**: So you mean first divide data into train and test set (hold out), rebalance train with Smote (for example) and then apply cross validation on this train set. What about instead, do oversample technique only during k fold iteration? let me explain better. Suppose i have unbalanced train, i apply 10-fold cross validation, so 9 groups will be used as train and 1 as test, what i mean is to apply oversample only at this time on the 9 train groups, just to reduce overfitting maybe. I hope i have been clear. i will wait your advices.  thanks  
  > **username_1**: @Alfonso You're quite right. That would be a better idea. My protocol leaks information from the validation set and yours does not.  
  
  ---
  
  