Up until now I’ve always dealt with splitting time series data into inputs of a specific length by running a sliding window of a particular size over each datapoint and saving each of the windows to a seperate directory to train a model on.
This is very time and memory consuming and means that it’s pretty long winded to try different sized inputs into my models.
I’m wondering if there’s a way I could do this just by using transforms with a data.Datasets object. I’ve managed to do it by taking random samples of a particular size from each datapoint but obviously this means that I end up with significantly less data than by my normal means as I’m only taking a small number of windowed samples from each datapoint rather than lots of windows by using the sliding window method.
For a number of cases the dataset is too large to fit into RAM so it’s not possible to load it all and just slide a window over the whole dataset and get the output using getitem
Cheers.