Solved by ptrblck in post #2 
              
              
                In your custom ConcatDataset you are returning a tuple containing samples of both datasets, but I think torch.utils.data.ConcatDataset would directly work.