Solved by ptrblck in post #2 
              
              
                Probably not the cleanest approach, but you could create the indices for both datasets first, shuffle them, wrap both datasets into a ConcatDataset, and use a Subset with the shuffled indices afterwards. 
Something like this might work: 
# create datasets
dataset1 = torch.utils.data.TensorDataset(to…