I think it might be useful for a lot of people to devise a roadmap of sorts when dealing with hdf5 files in combination with pytorch.
After digging deep into literally every thread on this board I draw the following conclusions that should be modified/extended as you see fit.
hdf5, even in version 1.10 does not support multiple process read, so that one has to find a solution to be able to use a worker number > 0 in the data loading process. But what is the best option here?


Load smaller datasets entirely from hdf5 to RAM prior to any training


Open the file once with a singleton of sorts within the getitem method as discussed



  
    
    
    DataLoader, when num_worker >0, there is bug 
  
  
    Yea, I’ve explored topic a bit and what I found is: 

With version 1.8 of HDF5 library working with HDF5 files and multiprocessing is a lot messier (not h5py! I mean HDF5 library installed on your system: https://unix.stackexchange.com/questions/287974/how-to-check-if-hdf5-is-installed). I highly recommend to update the library to 1.10 version where multiprocessing works better. I was only able to get h5py to work with “with” statement and this seems to give huge overhead, but I didn’t have time…
  




Use pandas tables, havent looked into it myself,


use zarr, I read about it, but also did not try it yet


can we find some common ground here ?