Body: <p>I have the following kind of data frame. These are just example:</p>
  
  <pre><code>A 1 Normal
  A 2 Normal
  A 3 Stress
  B 1 Normal
  B 2 Stress
  B 3 Stress
  C 1 Normal
  C 2 Normal
  C 3 Normal
  </code></pre>
  
  <p>I want to do 5-fold cross-validation and splitting the data using </p>
  
  <pre><code>skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=1)
  
  data = (ImageList.from_folder(PATH)
          .split_by_rand_pct(valid_pct=0.2)
          .label_from_folder()
          .transform(get_transforms(do_flip=True, flip_vert= True,max_zoom=1.1, max_rotate=10, max_lighting=0.5),size=224)
          .databunch()
          .normalize() )
  </code></pre>
  
  <p>It works great. It splits the data randomly which is expected. Though I want to keep the data points together in the training or validation, having the same value in column 1. So, all the A's would be in either the training or validation dataset, all the B's would be in the training or validation dataset, and so on.</p>
  
  <p>More info on my data:
  I have cell assay images which are labelled in three classes. Now, these images are big in size, so I split one image into 16 small, non overlapping tiles, to bring down the size to 224( optimal enough to feed into CNN). All these tiles have the same label as the original image. These tiles are the final input to the CNN. TO perform cross-validation, I need to keep the tiles of same image into one fold and set.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-24T04:02:14.077Z
Comments.Text: won't this ruin the validity of the validation ?
Comments.UserDisplayName: George White
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-24T15:34:23.903Z
Comments.Text: I agree with you in usual cases it would. I have different set of data.I edited in my question to explain it.
Comments.UserDisplayName: user1631306
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-24T15:37:40.670Z
Comments.Text: I think I found it a way to make it work using GroupKFold
Comments.UserDisplayName: user1631306
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-24T17:49:26.117Z
Comments.Text: @GeorgeWhite: Often it is the other way around - not performing a stratified split will ruin your validation, because you train on data that is correlated with validation data, and may think you have good generalisation but in fact it is bad. E.g. if your goal is to recognise emotions from images of faces in general (on an unseen person) and you have multiple pictures of same person, those pictures should stay together, either in training set or cv set or test set, but not in multiple sets.
Comments.UserDisplayName: Neil Slater
Tags: <python><cross-validation><scikit-learn>
Title: How can I split the data into training and validation sets such that entries with a certain value are kept together?
UserDisplayName: user1631306
ThreadText: # How can I split the data into training and validation sets such that entries with a certain value are kept together?
  
  (Asked by: username_0 on 2019-12-23)
  
  I have the following kind of data frame. These are just example:
  
  ```
  A 1 Normal
  A 2 Normal
  A 3 Stress
  B 1 Normal
  B 2 Stress
  B 3 Stress
  C 1 Normal
  C 2 Normal
  C 3 Normal
  
  ```
  
  I want to do 5-fold cross-validation and splitting the data using
  
  ```
  skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=1)
  
  data = (ImageList.from_folder(PATH)
          .split_by_rand_pct(valid_pct=0.2)
          .label_from_folder()
          .transform(get_transforms(do_flip=True, flip_vert= True,max_zoom=1.1, max_rotate=10, max_lighting=0.5),size=224)
          .databunch()
          .normalize() )
  
  ```
  
  It works great. It splits the data randomly which is expected. Though I want to keep the data points together in the training or validation, having the same value in column 1. So, all the A's would be in either the training or validation dataset, all the B's would be in the training or validation dataset, and so on.
  
  More info on my data:
  I have cell assay images which are labelled in three classes. Now, these images are big in size, so I split one image into 16 small, non overlapping tiles, to bring down the size to 224( optimal enough to feed into CNN). All these tiles have the same label as the original image. These tiles are the final input to the CNN. TO perform cross-validation, I need to keep the tiles of same image into one fold and set.
  
  **Tags:** python, cross-validation, scikit-learn
  
  > **Comments:**
  > **username_1**: won't this ruin the validity of the validation ?  
  > **username_0**: I agree with you in usual cases it would. I have different set of data.I edited in my question to explain it.  
  > **username_0**: I think I found it a way to make it work using GroupKFold  
  > **username_2**: @username_1: Often it is the other way around - not performing a stratified split will ruin your validation, because you train on data that is correlated with validation data, and may think you have good generalisation but in fact it is bad. E.g. if your goal is to recognise emotions from images of faces in general (on an unseen person) and you have multiple pictures of same person, those pictures should stay together, either in training set or cv set or test set, but not in multiple sets.  
  
  ---
  
  ## 0 answers
  
  