If you are using the GPU, you would need to synchronize the code before starting and stopping the timer via torch.cuda.synchronize(), since CUDA operations are executed asynchronously.
If you don’t manually synchronize, the next blocking operation will accumulate the timings from the previous ops, so that your results might give you the wrong bottlenecks.