Solved by ptrblck in post #4 
              
              
                I think you could create both Datasets and pass them to a custom Dataset, which concatenates the samples of both underlying Datasets. 
I’ve created a small example using ImageFolder: 

dataset1 = datasets.ImageFolder(
    root='YOUR_PATH',
    transform=transforms.ToTensor())
dataset2 = datasets.Ima…