Could you add a synchronization before the start of the timer?
Note that even though the operations should be similar, cuDNN etc. might chose specific algorithms which might be faster or slower than the counterpart.
Also, try to use torch.backends.cudnn.benchmark = True and let the operations run a few times before timing them.