Well I don’t think that the problem is:



 Carsten_Ditzel:

hdf5, even in version 1.10 does not support multiple process read


It does support it. The problem is that opened hdf5 file can’t be pickled as I mentioned here:



DataLoader, when num_worker >0, there is bug


Because an opened HDF5 file isn’t pickleable and to send Dataset to workers’ processes it needs to be serialised with pickle, you can’t open the HDF5 file in __init__ . Open it in __getitem__ and store as the singleton! . Do not open it each time as it introduces huge overhead.


Work around proposed here: DataLoader, when num_worker >0, there is bug 217 works perfectly fine for me for really big datasets.
Also, I don’t think there is a silver bullet for this problem. What to use is specific to a use case.