I think that the easiest solution would be to use one DataLoader per Dataset.
In fact, if you use only one dataloader from both tasks, the problem is that one batch can be composed of images from d1 and from d2. It is ok to have images from both datasets for your main feature extractor but it becomes painful when you have to separate the produced features for tasks A and B since splitting the batch will produce batches of different lengths, with eventually empty batches, etc.
If you really want to use only one unique dataloader, you can have a look at ConcatDatasets 47.