Could you try with num_workers=0 instead? I think multiprocessing breaks your intended behaviour.
The correct approach for this kind of problem is using a custom sampler, that will give you indices for all batches. This way, the dataloader can have multiple processes and be faster, it will ask the sampler (only one instance) for indices, and transfer those indices to the dataset, therefore loading the correct data.
More detailed explanations in the doc 1.