Answers.Body: <p>I don't have much information about your model and your data. But my best guess is that your dataset doesn't have the same distribution everywhere, and that without shuffling it before doing the split, you create subsets of data that are statistically very different i.e. their intrinsic distribution might differ a lot. When you shuffle before splitting, you mitigate this effect because you distribute your data (which might follow divergent patterns according to timestamps or any other feature) in both sets.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-15T08:49:54.197Z
Answers.Text: That could definitely make sense. I had a similar thought and compared the distribution of the features (in fact a histogram since they are categorical) between train and test. The plots look slightly different but not tremendously. However, I have to admit that they could be more insightful. Do you have an idea how to best approach such a statistical comparison and what metrics to look out for? A very simple approach could just be DataFrame.describe().
Answers.UserDisplayName: NicFit_88
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-13T12:47:59.950Z
Answers.LastActivityDate: 2022-08-13T12:47:59.950Z
Answers.UserDisplayName: Dime
Body: <p>I am currently working with a very deep NN (200mio. to 350mio. params). My data set is roughly of shape (2mio, 350), i.e. 2mio samples and 350 features. In fact, the features are time series. As input to the NN I just pass the current state (1 time step), however that state is derived with some sort of scaling from past states.</p>
  <p>Now I made a couple strange observation during training, I can't explain at all:</p>
  <ol>
  <li>When I shuffle the whole data set and then do train/test split, the classification performance is extremely good, ~0.93 accuracy (3 classes) and identically generalizes onto the test set during the whole training</li>
  <li>When I split the data into train/test and just shuffle train, the performance is less on train, but still acceptable (~0.75 accuracy), but performance on test falls off to ~0.36 accuracy</li>
  <li>When I just split the data without shuffling, the train performance further drops to ~0.5 and test performance to ~0.3 (which is worse than randomly guessing)</li>
  </ol>
  <p>What is going on here? I checked if I might have introduced some data leakage or lookahead bias in my data set manipulations, but couldn't find anything.</p>
  <p>Anybody has an idea what is going on here?</p>
  <p>Cheers</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-14T23:14:08.697Z
Comments.Text: How are you shuffling your data? For time series not every shuffling is conceptually right, see [this answer](https://ai.stackexchange.com/questions/36407/can-i-shuffle-data-for-delivery-duration-forecast-problem/36409#36409) for an explanation
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-19T16:30:34.213Z
Comments.Text: what is mio mean?
Comments.UserDisplayName: JobHunter69
Tags: <neural-networks><tensorflow><keras><transformer>
Title: Shuffling vs Non-shuffling train/test set yields drastically different results
UserDisplayName: NicFit_88
ThreadText: # Shuffling vs Non-shuffling train/test set yields drastically different results
  
  (Asked by: username_0 on 2022-08-12)
  
  I am currently working with a very deep NN (200mio. to 350mio. params). My data set is roughly of shape (2mio, 350), i.e. 2mio samples and 350 features. In fact, the features are time series. As input to the NN I just pass the current state (1 time step), however that state is derived with some sort of scaling from past states.
  
  Now I made a couple strange observation during training, I can't explain at all:
  
  1. When I shuffle the whole data set and then do train/test split, the classification performance is extremely good, ~0.93 accuracy (3 classes) and identically generalizes onto the test set during the whole training
  2. When I split the data into train/test and just shuffle train, the performance is less on train, but still acceptable (~0.75 accuracy), but performance on test falls off to ~0.36 accuracy
  3. When I just split the data without shuffling, the train performance further drops to ~0.5 and test performance to ~0.3 (which is worse than randomly guessing)
  
  What is going on here? I checked if I might have introduced some data leakage or lookahead bias in my data set manipulations, but couldn't find anything.
  
  Anybody has an idea what is going on here?
  
  Cheers
  
  **Tags:** neural-networks, tensorflow, keras, transformer
  > **Comments:**
  > **username_1**: How are you shuffling your data? For time series not every shuffling is conceptually right, see [this answer](https://ai.stackexchange.com/questions/36407/can-i-shuffle-data-for-delivery-duration-forecast-problem/36409#36409) for an explanation  
  > **username_2**: what is mio mean?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-08-13. Score: 1
  
  I don't have much information about your model and your data. But my best guess is that your dataset doesn't have the same distribution everywhere, and that without shuffling it before doing the split, you create subsets of data that are statistically very different i.e. their intrinsic distribution might differ a lot. When you shuffle before splitting, you mitigate this effect because you distribute your data (which might follow divergent patterns according to timestamps or any other feature) in both sets.
  
  > **Comments:**
  > **username_0**: That could definitely make sense. I had a similar thought and compared the distribution of the features (in fact a histogram since they are categorical) between train and test. The plots look slightly different but not tremendously. However, I have to admit that they could be more insightful. Do you have an idea how to best approach such a statistical comparison and what metrics to look out for? A very simple approach could just be DataFrame.describe().  
  
  ---
  
  