I am wondering if I can modify __get_item__ in Dataset to accept multiple indices instead of one index at a time to improve data loading speed from disk using H5 file.
My dataset looks something like this
class HDFDataset(Dataset):
    def __init__(self, path):
        self.path = path
        
    def __len__(self):
        return self.len

    def __getitem__(self, idx):
        hdf = h5py.File(path, 'r')
        data = hdf['data']
        X = data[idx,:]
        hdf.close()
        return X

dset = HDFDataset(path)
I also have a custom batch sampler
def chunk(indices, chunk_size):
    return torch.split(torch.tensor(indices), chunk_size)


class BatchSampler(Sampler):
    def __init__(self, batch_size, dataset):
        self.batch_size = batch_size
        self.dataset = dataset
    
    def __iter__(self):
        #some function
        return iter(list_of_list_idx)
    

Sample output from BatchSampler looks as below. Each list represents a batch and each values indicate the indices in a batch
batch_sampler = BatchSampler(batch_size, dataset)
for x in batch_sampler:
      print(x)


[12, 3, 8, 6, 17]
[7, 9, 1, 19, 18]
[13, 4, 2, 5, 14]
[0, 3, 10, 11, 20]

Dataloader looks like
train_dataloader = DataLoader(dset, num_workers=8, batch_sampler=batch_sampler)
This approach works fine for me but data loading takes time as __get_item__ loads one index at a time from disk.
Since I already know the indices I need to load in each batch using BatchSampler, is there a way I can load entire batch at once in dataset
for e.g if batch 1 indices is
batch_idx = [12, 3, 8, 6, 17]
if __get_item__ can accept list of indices than single indice, something like below
def __getitem__(self, batch_idx):
        hdf = h5py.File(path, 'r')
        data = hdf['data']
        X = data[batch_idx,:]
        hdf.close()
        return X

Solution 1: Since I already know the indices in each batch I can just load the data to model as tensors, however I won’t be able to utilize num_workers parameter in DataLoader to speed up.
If there is a way to load data in chunks using Dataset & DataLoader, it would solve my issue.
Appreciate any suggestions.