CUDA operations are executed asynchronously, so you would need to synchronize the code before starting and stopping the timer.
Also, note that the very first CUDA operation initializes the CUDA context etc. and is thus slower.
You could use the torch.utils.benchmark utilities to profile specific operations, which would make sure to properly add warmup iterations and synchronizations.
If you are seeing a performance regression in the model execution, please post the model definition (if possible), so that we could test it with the latest cudnn release and check the performance on this GPU.
That being said, note that we are currently facing an issue in the PyTorch binaries with statically linked cudnn, which removes xmma kernels as described here 15, and we are still working on forcing these kernels into the binaries.