As your timing differences are in the range of values I just measured to verify a fix of https://github.com/pytorch/pytorch/issues/25010 17 , you might want to try setting pin_memory=False in the main.py and see if that improves things for your slow cases…
My timings running the mnist train for the default 10 epochs. +/- 1 second is in the noise, there is obviously a big issue in 1.2 (and I see it in 1.1). I don’t see it in 1.0. Interestingly there is a diff btw 1.0 and 1.2 as 57 vs 63-64 is not in the measurement noise. I don’t have numbers across different cuda versions or pip vs conda though.
PyTorch 1.2 Conda, Python 3.7.3, Cuda 10.0.130, CUDNN 7.6.2
pin_memory=True, num_workers=1, no fix applied: 131.03s
pin_memory=True, num_workers=1, fix applied: 63.5s
pin_memory=False, num_workers=1: 63.25s
pin_memory=True, num_workers=0, no fix applied: 66.70s
pin_memory=True, num_workers=0, fix applied: 64.42s
pin_memory=False, num_workers=0: 64.03s

PyTorch 1.0, Conda, Python 3.6.8, Cuda 10.0.130, CUDNN 7.4.1
pin_memory=True, num_workers=1: 57.83
pin_memory=False, num_workers=1: 57.48
pin_memory=True, num_workers=0: 57.32
pin_memory=False, num_workers=0: 56.95