Thank you, that really explains the bad behavior of my neural net. So, the obvious follow-up question would be is there a way to get my desired behavior? I was thinking concatenating DataLoaders instead since the DataLoader for each Dataset would contain images from that particular dataset online batch-wise. After concatenating I wouldn’t have to worry about a mix-up. Anything like that possible in Pytorch?