In this case, you can write a custom Dataset (http://pytorch.org/docs/master/data.html#torch.utils.data.Dataset 168) that for a given index, returns the indexed data from the first dataset and samples a second one.