Thanks for the heads up @ptrblck!
Changed the two bools as suggested, added an extra synch call before stopping the timer, add an additional backward pass before timing for profiling for fp32 and a fwd/backward autocast pass. New timing with the additional synch call for the Pytorch 1.5 server:

So the timings went down - due to changing the bool values, as suggested - but still the wrong (intuitively) order.
@seungjun - thanks as well for the tip. I tried with Resnet18 and Resnet50 and the timings are even, which seems more reasonable, though is still a bit surprising.


The 1.6 server is currently occupied, will be able to update tomorrow. Unfortunately, I’m using a cloud provider that only has certain docker images - in this case:

pytorch:1.5.0-cuda10.1-cudnn7-devel
pytorch:1.6.0-cuda10.1-cudnn7-devel
and the two @ptrblck mentioned aren’t available :(.