The tutorials (such as  this one 199) show how to use torch.utils.data.Dataset to efficiently load large image datasets (lazy loading or data streaming). This is easily applied to images because they usually exist as a folder containing separate files (each sample exists as its own file), and so it’s easy to load just a single image at a time (usually with a csv serving as a manifest that “points” to each image by storing the filename, instead of the pixel data).
However, for other types of data, sometimes we receive a dataset as a gigantic pandas dataframe (maybe stored in an HDF5 file) or as a large numpy .npy file. If these are also large (larger than my memory), how can I use torch.utils.data.Dataset to efficiently stream it?
Does it boil down to stripping out each row (a sample) in the dataframe/matrix and saving that as a separate file, and creating a csv manifest of these files?