In general, I write my own Dataset class that inherits from the PyTorch Dataset and it handles all the logic of what data and labels to feed to the network when. Then the PyTorch Data Loader doesn’t have to know about any of that, it just loads pairs.
Disclaimer: I didn’t read the code, so I’m not sure precisely what the problem is.