Interesting, after investigating the torch versions, it turns out that the GTX machine had torch version == 0.4.1 and the RTX had version == 1.4.0. When I upgraded the GTX version upwards to 1.4.0, I was able to reproduce the bad performance of the RTX.
Why is it the case?
In terms of synchronizing the times, I’m just measuring the time it takes to complete a forward iteration for a batch and  also measuring the time that it takes the dataloaders to get the data. So the difference between the two is the time it takes to perform a forward pass in the data. In any case, upgrading the pytorch version makes my epochs go from 5 minutes to complete to 80 minutes.