there’s no reason the validation set should run out of memory, that’s weird.
I think you should write a custom dataset for the following reason:

what you want is class-balancedness. i.e. you want network to see same-class / different-class with equal probability. Your two-dataloader solution will sample same-class only with 1/M probability and different-class with (M-1)/M probability.

Look at taking an existing Dataset and modifying it for your purpose: https://github.com/pytorch/vision/tree/master/torchvision/datasets 16