Hi,
I was wondering whether it is possible to resume iterating through a dataloader from a checkpoint.
For example:
    dataloaders_dict = {phase: torch.utils.data.DataLoader(datasets_dict[phase], batch_size=args.batch_size, num_workers=args.num_workers, shuffle=False) for phase in ['train']} # make sure shuffling is false incase you restart

if os.path.isdir(args.batchidx_checkpoint):
            checkpoint = torch.load(args.batchidx_checkpoint + 'batches.pt')
            batch_idx = checkpoint['batch_idx'] + 1
     
        else:
            os.makedirs(args.batchidx_checkpoint)
            batch_idx = 0

for batch_idx, (inputs, labels) in enumerate(dataloaders_dict['train']):
            #print(batch_idx)
            # do some feature extraction and save feature arrays in hdf5 file  


I have seen some examples mentioning random sampler, but was wondering if it is really necessary…
I also have about 5 million images to go through, so it may be a large dataset to iterate through. My end goal is to save features for each image using a pretrained network, thus no training is really required. I just want to resume feature extraction from a checkpoint batch id…
Cheers,