I have a large dataset stored remotely in a parquet file, which I would like to read and train without storing the entire file in memory. The dataset is multivariate timeseries for clients and goes in the format:
N*Q*M
# N: number of users (50000)
# Q: sequence length (365)
# M: number of features (20)

In a dataframe this would be represented such that rows for a given user are consecutive, but there is also an additional user index column.
How do I read and batch such a large file without reading it all in memory, assuming that I set a batch size of 16 (the number of clients is 16, so the total number of sequences in each batch is 16 * 365 = 5840) ?
Splitting into a smaller number of files and using torch.utils.data.Dataset is not really an option since reading many files from a remote directly in every epoch is very expensive. Therefore, I have been trying with torch.utils.data.IterableDataset but I haven’t figured a way for reading certain records of the file. (This is all assuming I am using no workers, as increasing the number of workers for distributed reading is a whole new issue.)