Body: <p>My goal is to classify windows of multivariate timeseries data into a positive and a negative class. When I construct sliding time windows and randomly shuffle them into the train and test sets, I get surprisingly high test scores, leading me to suspect data leakage somewhere in my pipeline. Should I first be splitting the data at a chosen timestamp into train and test sets and then building sliding windows for the two sets separately? My understanding is that the rule-of-thumb states never to train with timeseries data from the &quot;future&quot;, which is what happens with the way I currently do it.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-14T07:32:08.173Z
Comments.Text: A timeseries is time dependent so randomly shuffle maybe is not the best option (same for a random split). Check this [question](https://stackoverflow.com/questions/74025273/is-train-test-splitshuffle-false-appropriate-for-time-series)
Comments.UserDisplayName: Robert
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-14T17:28:27.310Z
Comments.Text: Thanks! Would this be true if my windows are non-overlapping though?
Comments.UserDisplayName: bonzo_pippinpaddle
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-15T10:57:57.267Z
Comments.Text: I'm not expert, but probably you can not assume independence between windows, despite beeing disjoint.
Comments.UserDisplayName: Robert
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-23T01:17:05.550Z
Comments.Text: @Robert Why is shuffling here bad? It seems like he's trying to train on random independent windows of time.
Comments.UserDisplayName: JobHunter69
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-23T23:24:32.523Z
Comments.Text: @JobHunter69, that's right. I am training on random independent windows. This problem also made me curious about the i.i.d. assumption in general. It makes sense to me that randomly split time windows may not be i.i.d. since the underlying process generating the data is identical, and, therefore, adjacent time points are correlated. But then is that not true in all ML? Since we are always assuming the training and validation/test data comes from roughly identical distributions, "adjacent" points would also not be i.i.d.
Comments.UserDisplayName: bonzo_pippinpaddle
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-24T06:53:17.667Z
Comments.Text: @bonzo_pippinpaddle but shuffling makes the adjacent points iid right?
Comments.UserDisplayName: JobHunter69
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-26T15:26:18.217Z
Comments.Text: Still thinking that "time" is key.  Probably the solution here is in time-series or stats theory. Anyway the assumption of i.i.d. is always curious, many times done in stats but few times happens on real life.
Comments.UserDisplayName: Robert
Tags: <classification><time-series>
Title: What should you consider when splitting timeseries data into train and test sets?
UserDisplayName: bonzo_pippinpaddle
ThreadText: # What should you consider when splitting timeseries data into train and test sets?
  
  (Asked by: username_0 on 2024-02-14)
  
  My goal is to classify windows of multivariate timeseries data into a positive and a negative class. When I construct sliding time windows and randomly shuffle them into the train and test sets, I get surprisingly high test scores, leading me to suspect data leakage somewhere in my pipeline. Should I first be splitting the data at a chosen timestamp into train and test sets and then building sliding windows for the two sets separately? My understanding is that the rule-of-thumb states never to train with timeseries data from the "future", which is what happens with the way I currently do it.
  
  **Tags:** classification, time-series
  
  > **Comments:**
  > **username_1**: A timeseries is time dependent so randomly shuffle maybe is not the best option (same for a random split). Check this [question](https://stackoverflow.com/questions/74025273/is-train-test-splitshuffle-false-appropriate-for-time-series)  
  > **username_0**: Thanks! Would this be true if my windows are non-overlapping though?  
  > **username_1**: I'm not expert, but probably you can not assume independence between windows, despite beeing disjoint.  
  > **username_2**: @username_1 Why is shuffling here bad? It seems like he's trying to train on random independent windows of time.  
  > **username_0**: @username_2, that's right. I am training on random independent windows. This problem also made me curious about the i.i.d. assumption in general. It makes sense to me that randomly split time windows may not be i.i.d. since the underlying process generating the data is identical, and, therefore, adjacent time points are correlated. But then is that not true in all ML? Since we are always assuming the training and validation/test data comes from roughly identical distributions, "adjacent" points would also not be i.i.d.  
  > **username_2**: @username_0 but shuffling makes the adjacent points iid right?  
  > **username_1**: Still thinking that "time" is key.  Probably the solution here is in time-series or stats theory. Anyway the assumption of i.i.d. is always curious, many times done in stats but few times happens on real life.  
  
  ---
  
  ## 0 answers
  
  