@ptrblck You understood it correctly.
I actually found a way, I dump both the datasets (dataset1: CIFAR10, dataset2: CIFAR10 variant) using torch.save and load it back like this;
img = torch.load(filename_images)
lab = torch.load(filename_labels)
        
xs = torch.stack(img)
xs = xs.squeeze(1)
ys = torch.Tensor(lab)
dataset = TensorDataset(xs, ys)

this dataset works perfectly normal in torch.utils.data.DataLoader. And the new concatenated data is already shuffled above before dumping but I do shuffle it again when I load it for training.