Currently, I have 500+ Pickle files that hold time-series data in the form of data frames, where each data frame represents a single day. Each of these data frames hold ~10,000 rows of data and ~500 features. I want to use the data and feed it through an LSTM model; however, loading the entire data set and doing a loop to create (input, output) tuples is too memory intensive. In addition, since 1 Pickle file does not represent a single LSTM input, I cannot simply load a single file in the get_item function as a solution.
How do I create a DataLoader for this situation? Would I have to load all of the data at once due to it being time-series data? Would I have to change the data format so that each file represents a single sample (which would also have the issue of just having too many files). Any help is appreciated!