Hi,
So I added:
import torch.multiprocessing as mp
mp.set_start_method('fork')

Since I am using Linux and no longer get an error in regards to not being able to pickle the file. However, when I increase the number of workers, I still get  OSError: Can't read data (wrong B-tree signature)
I investigated and it does appear that the multithreading mixes up some of the values. For example, empty arrays being returned or even labels when only asking for paths to be printed.
The main issue is that my GPU doesn’t seem to get utilised. But the memory usage is high. I’m not really sure what else there can be done. I have tried a number of fixes, none of them appear to be useful and sometimes make everything much worse. I have even set pin_memory to True.
This is the updated dataclass that appears to retrieve data fairly quickly without deployment on the GPUs:
import torch.multiprocessing as mp
mp.set_start_method('fork')

from torch.utils import data
import h5py

class Features_Dataset(data.Dataset):
    def __init__(self, archive, phase):
        self.archive = h5py.File(archive, 'r', libver='latest', swmr=True)
        assert self.archive.swmr_mode
        self.labels = self.archive[str(phase) + '_labels']
        self.data = self.archive[str(phase) + '_all_arrays']
        self.img_paths = self.archive[str(phase) + '_img_paths']

    def __getitem__(self, index):
        datum = self.data[index]
        label = self.labels[index]
        path = self.img_paths[index]
        return datum, label, path

    def __len__(self):
        return len(self.data)

    def close(self):
        self.archive.close()

if __name__ == '__main__':
    train_dataset = Features_Dataset(archive= "featuresdata/train.hdf5", phase= 'train')
    trainloader = data.DataLoader(train_dataset, num_workers=2, batch_size=2)
    print(len(trainloader))
    for i, (data, label, path) in enumerate(trainloader):
        print(path)



Baffling. I’m sure people must be feeding in large HDF5 datasets into their networks!