Hello all, I have a dataset that requires to use h5 files. The dataset size has 1000 images with the total size of 100GB. I have two options:

Save each image to an hdf5 file, then I have total 1000 hdf5 file. In dataloader, I will call

class h5_loader(data.Dataset):
    def __init__(self, file_path):
	    self.file_list = [f for f in glob.glob(os.path.join(file_path, '*.h5'))]
    def __getitem__(self, index):
	    h5_file = h5py.File(file_list[index])
        data = h5_file.get('data')


Save all images into a single hdf5 file, then dataloader can call

class h5_loader(data.Dataset):
    def __init__(self, file_path):
        h5_file = h5py.File(file_path)
        data = h5_file.get('data')
    def __getitem__(self, index):
		...


Which option should I use to speed up data loader time? The first way load each hdf4 in __getitem__ function, while the second way load a single (combined of 1000 images) in the __init__ function.