Haha, no he might be right!
It would be interesting to hear his opinion and I might be completely wrong. 
In this use case, I think splitting the Datasets before wrapping them in DataLoaders would be clearer than using a custom sampler, collate_fn or other methods to get the corresponding samples from a single DataLoader.