Solved by ptrblck in post #2 
              
              
                Your approach sounds reasonable. 
I think you could change SiameseDataset a bit and just sample from the ConcatDataset as shown here: 
class SiameseDataset(Dataset):
    def __init__(self, dataset):
        super().__init__()
        self.dataset = dataset

    def __getitem__(self, index):
        …