Answers.Body: <p>Most time series contain time-dependent information. That means that the time series has order. The temporal information, therefore, cannot be randomly sampled without losing some pertinent information.</p>
  <p>By splitting a time series, we retain the temporal information to generalize a model, such as a classifier or a regressor, while setting aside a smaller time series to test. A simple split, such as 80%/20% can be done or something fancier such as K-Fold validation can be done.</p>
  <p>The hope is that the information in the training dataset persists and appears in the testing dataset. It is possible that an event changes the time series enough, and coincidentally at the split time, such that the testing dataset is of no value. If that happens, you have little option but to wait until there is enough information to justify a new split while trimming off the data before the event.</p>
  <p>The following paper reviews the many strategies to splitting time series data:</p>
  <ul>
  <li>Cerqueira, V., Torgo, L., &amp; Mozetič, I. (2020). Evaluating time series forecasting models: an empirical study on performance estimation methods. Machine Learning, 109(11), 1997–2028. <a href="https://doi.org/10.1007/s10994-020-05910-7" rel="nofollow noreferrer">https://doi.org/10.1007/s10994-020-05910-7</a></li>
  </ul>
  <p>The paper is also available as a PDF <a href="https://arxiv.org/pdf/1905.11744" rel="nofollow noreferrer">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-01T00:13:16.983Z
Answers.LastActivityDate: 2023-01-01T00:19:18.090Z
Answers.LastEditDate: 2023-01-01T00:19:18.090Z
Answers.UserDisplayName: Brian O'Donnell
Body: <p>Some of you may be familiar with the <a href="https://scikit-learn.org/stable/modules/cross_validation.html#time-series-split" rel="nofollow noreferrer">unusual split scheme</a> used for time-series data. In short, there is a saying that one should only consider a split where the training set comes prior to the testing set (in terms of index or <code>timedate</code>), as otherwise we essentially use future data to infer.</p>
  <p>Namely, given the dataset <span class="math-container">$\mathcal{D}=\{(x_1,y_1),...,(x_n,y_n)\}$</span>, a viable split may look like
  <span class="math-container">\begin{align}
   train=\{(x_1,y_1),...,(x_j,y_j)\}\subseteq\mathcal{D}\\
   test=\{(x_{j+1},y_{j+1}),...,(x_k,y_k)\}\subseteq\mathcal{D}
  \end{align}</span>
  for some <span class="math-container">$j\leq k\leq n$</span></p>
  <p>My question is - are there some cases where random splitting is O.K in time series? Also, what is the main problem with random sampling?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-31T23:54:49.237Z
Comments.Text: Please clarify what you mean by 'random splitting'. It sounds like you want to choose j randomly. The standard practice is to split data with 80% for training and 20% for testing. When the data has a low number of observations many people will choose 90% for training and 10% for testing.
Comments.UserDisplayName: Brian O'Donnell
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-01T00:01:36.957Z
Comments.Text: random as in k fold, for example
Comments.UserDisplayName: Hadar Sharvit
Tags: <time-series><k-fold-cv>
Title: Should we always use the usual no leakage train-val-test splt in time series?
UserDisplayName: Hadar Sharvit
ThreadText: # Should we always use the usual no leakage train-val-test splt in time series?
  
  (Asked by: username_0 on 2022-12-31)
  
  Some of you may be familiar with the [unusual split scheme](https://scikit-learn.org/stable/modules/cross_validation.html#time-series-split) used for time-series data. In short, there is a saying that one should only consider a split where the training set comes prior to the testing set (in terms of index or `timedate`), as otherwise we essentially use future data to infer.
  
  Namely, given the dataset $\mathcal{D}=\{(x\_1,y\_1),...,(x\_n,y\_n)\}$, a viable split may look like
  \begin{align}
  train=\{(x\_1,y\_1),...,(x\_j,y\_j)\}\subseteq\mathcal{D}\\
  test=\{(x\_{j+1},y\_{j+1}),...,(x\_k,y\_k)\}\subseteq\mathcal{D}
  \end{align}
  for some $j\leq k\leq n$
  
  My question is - are there some cases where random splitting is O.K in time series? Also, what is the main problem with random sampling?
  
  **Tags:** time-series, k-fold-cv
  > **Comments:**
  > **username_1**: Please clarify what you mean by 'random splitting'. It sounds like you want to choose j randomly. The standard practice is to split data with 80% for training and 20% for testing. When the data has a low number of observations many people will choose 90% for training and 10% for testing.  
  > **username_0**: random as in k fold, for example  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-01-01. Score: 2
  
  Most time series contain time-dependent information. That means that the time series has order. The temporal information, therefore, cannot be randomly sampled without losing some pertinent information.
  
  By splitting a time series, we retain the temporal information to generalize a model, such as a classifier or a regressor, while setting aside a smaller time series to test. A simple split, such as 80%/20% can be done or something fancier such as K-Fold validation can be done.
  
  The hope is that the information in the training dataset persists and appears in the testing dataset. It is possible that an event changes the time series enough, and coincidentally at the split time, such that the testing dataset is of no value. If that happens, you have little option but to wait until there is enough information to justify a new split while trimming off the data before the event.
  
  The following paper reviews the many strategies to splitting time series data:
  
  * Cerqueira, V., Torgo, L., & Mozetič, I. (2020). Evaluating time series forecasting models: an empirical study on performance estimation methods. Machine Learning, 109(11), 1997–2028. <https://doi.org/10.1007/s10994-020-05910-7>
  
  The paper is also available as a PDF [here](https://arxiv.org/pdf/1905.11744).
  
  
  ---
  
  