How to deal with the problem of correlated samples from time series in machine learning We know that most of the machine learning algorithms assume the samples are i.i.d. However if the samples are from time series (like ARMA in the prediction of time series), then they are highly correlated. Then how to deal with such correlated samples? Note that, here I mean the correlation between samples not the correlation between features.

*

*Example 1: we have a time series $\{x_1,\cdots,x_T\}.$ Then we use three times to predict the forth time to construct the training samples:

$$(X_1 = (x_1,x_2,x_3), y_1 = x_4), (X_2 = (x_2,x_3,x_4), y_2 = x_5), \cdots.$$
Then the training samples $(X_i,y_i)$ must be highly correlated, which is contradict to the assumption of i.i.d. samples for most machine learning algorithms.


*Example 2: we have a time series $\{x_1,\cdots,x_T\}$ and the corresponding label series  $\{y_1,\cdots,y_T\}.$ Then the training samples:

$$(x_1, y_1), (x_2, y_2), \cdots$$
is also highly correlated.
I have the following guess:

*

*In principle, machine learning algorithm is not a proper way to deal with the time series?


*The assumption of i.i.d. samples is actually not necessary for the machine learning algorithm?
I think it is very basic questions. Can anyone give me some references?


Then how to deal with such correlated samples?

This boils down to how these systems are trained, i.e., sequence-to-sequence training. If we do that, the correlation structure is preserved, such as LSTMs or any RNN's training.

machine learning algorithm is not a proper way to deal with the time series?

Correlations are encoded in the models such as Gaussian process regression requires covariance matrix i.e., Kernel matrices, these matrices encodes training sample's correlation structure. See Usual covariance functions.

The assumption of i.i.d. samples is actually not necessary for the machine learning algorithm?

Time-series modelling may not be classified as "vanilla supervised learning". So from statistical learning theory perspective, they are special class of models with added assumptions. So, how time-series model's set up will not justify the argument of i.i.d. samples is actually not necessary for the machine learning algorithm.
