I’ve used ConcatDataset  on different dataset sizes. For example, if you’d like to concatenate training and validation datasets you can do this:
train_set = torch.utils.data.ConcatDataset( [train_set, validate_set])
You’ll need to make sure that both datasets return the same items, else, you’ll have a problem.