The reason why dataloader cannot fetch some batch in the middle is once the dataloader is created, it’s prefetching the data according to the internal order inside the dataloader.
So if you want to get i-th batch or get a random batch, you have to tell the loader what you want(so it’s not actually ith, or random)
The method I propose only works on master, https://pytorch.org/docs/master/data.html?highlight=subset#torch.utils.data.Subset 182.
The version for 0.4 could be using https://pytorch.org/docs/stable/data.html?highlight=subset#torch.utils.data.sampler.SubsetRandomSampler 176.
so, something like
sampler = SubsetRandomSampler(list(range(i*batch_size, (i+1)*batch_size)))
train_loader = DataLoader(cifar10.Cifar10(train=True,  image_transform=cifar10.transformer), sampler=sampler, shuffle=False, batch_size=BATCH_SIZE, num_workers=BATCH_SIZE)

This is the simplest way I can think of.