Could you profile the data loading loop separately (remove the actual training) and compare the epoch time for the ConcatDataset and the “zip approach” using a different number of workers?
Note that too many workers might also decrease your performance.
PS: Not sure if that’s your use case, but do you really need DoubleTensors? You will get a performance hit, if you are using a GPU, as float64 is slower than float32.