Could you check, if you are running out of host memory and are using the swap?



 Weixin_Jin:

I guess I can load the dataset into memory one file by one file. Is it possible?


Yes, this would be possible, but you would have to consider how and if shuffling should be working.
E.g. if you are lazily opening each file in the __getitem__ method, I would expect the performance to be really bad.
On the other hand, you could implement a logic, which opens a single numpy file, shuffles the indices, reads all samples, and loads the next file.
In case you are indeed running out of memory, you could try to use np.memmap, which doesn’t read the complete array, but allows you to index it from your disc.