CUDA operations are executed asynchronously, so you would need to synchronize the code via torch.cuda.synchronize() manually before starting and stopping the timers.
Alternatively, you could also use torch.utils.benchmark, which will add warmup iterations and synchronize the code for you.