If the dataset that you are using is an IterableDataset then I don’t believe that converting the DataLoader for that dataset should be messing up distributed sampling. Since it looks like it is working when using one dataloader for you, isn’t a possible workaround to try combining the two different tasks into one custom data loader and implement your own __next__() with the task probabilities.