I have a directory with huge parquet files and have been using fastparquet to read in the files, which works fine.
I want to extend the Dataset class to read them lazily and hope to have a better GPU utilisation.
Here are a few questions regarding the Dataset class:

The len method: Should it return the number of training instances or the number of parquet files in the directory?
The getitem method: Should it return just a single training row (ndarray of shape (1, num_features)) or can it also return a matrix of rows (ndarray of shape (num_rows_in_a_parquetfile, num_features))?

If it can return a matrix, how to feed it into Dataloader class?
Almost all that I have understood reading the forums is that people use it to load images (one instance per file). Not sure how does this work out for reading huge files with N number of rows.
Thanks