I have large hdf5 database, and have successfully resolved the thread-safety problem by enabling the SWARM feature of hdf5. However, using multiple worker to load my dataset still not achieve normal speed. Typically, I observe the GPU utility circularly rise up to 100%, then drop down to 1%.
Here is my dataset code (seems very naive):
class HDF5Dataset(Dataset):
    """
    Args:
        h5data (HDF5 dataset): HDF5 dataset object
    """
    def __init__(self, h5data):
        self.h5data = h5data
        
    def __getitem__(self, index):
        return self.h5data[index, ...]

    def __len__(self):
        return len(self.h5data)

then I use the above code this way
f = h5py.File('Mydata.h5', 'r', libver='latest', swmr=True)
h5data = f['Data']
dataset = HDF5Dataset(h5data)
train_loader = DataLoader(dataset, batch_size=4, num_workers=4)

how can i solve this problem? what’s the best pratice to load large hdf5 datasets in pytorch? or I should follow What’s the best way to load large data? to migrate my data into lmdb?