Is shuffling timeseries data, then separating into training/testing sets a form of data leakage? I am building a multiple regression MLP (Multi-Layer Perceptron), the input is 8 weather variables collected from October-February, and the output is another weather variable.
The assumption is that the 8 variables can be used to predict the output variable. Thus far, the accuracy has been exceptional. Suspiciously exceptional... which puts doubt in my mind.
In the dataset, the datapoints are organized by timestamp, beginning October and ending in February.
The potential problem is this:
Is shuffling the datapoints and then splitting into training/validation/testing datasets a form of data leakage?
I've made sure to only z-scale the data based on the training data, using sklearn.preprocessing.StandardScaler()
When the data was not shuffled beforehand, the model performed less accurately on the testing and validation data. Since the data was not shuffled in this case, the testing data was only data from February. Perhaps this is because the model never "saw" typical weather conditions from February.
After training a model on the shuffled data, the Mean Squared Error for the validation and testing sets nearly halved.

What distinguishes time series data from other types of data is that data are collected over time (e.g. hourly, daily, weekly, monthly, etc.) and there is correlation between adjacent observations so the order of the data matters.
If you're arbitrarily shuffling time series, you're breaking the correlation structure inherent in the data, and whatever model you fit won't generalize well to your series in the future regardless of how good your error metrics look using an arbitrary split of the data. You may wish to consider validation methods designed for time series, such as time series cross-validation which evaluates your model on a rolling basis:

If you're a Python user, you may also wish to consult the documentation for sklearn or prophet).
