If you are timing cuda ops, you should add a torch.cuda.synchronize() before starting and stopping the timer.
The first cuda call needs some time to init CUDA, thus your timing might measure this time as well.#
Also CUDA ops are called asynchronously, so that your main thread can continue its execution while the GPU is busy.
Could you add it to your code and run it again?