In your scripts you are rightfully synchronizing before starting the timer, but no synchronization is used when you are stopping the timer.
Thus your timing might be wrong and in fact you might profile the PyTorch overhead and the kernel launch times.
Add syncs before starting and stopping the timer and execute the real workload (forward/backward) inside a loop to get more stable results.
Also, setting torch.backends.cudnn.benchmark = True and cudnn.deterministic = False might also help, as cudnn will profile different algos and will select the fastest one.
Note that this setup would increase the first iteration time (for forward and backward) significantly, as the profiling would be executed, so you should add warmup iterations.
I would also recommend to use the CUDA10.2 binaries, which ship with cudnn7.6.5.32 or the nightly CUDA11.0 binaries, which come now with cudnn8.0.4.30.