CUDA operations are asynchronous so you would need to synchronize the code via torch.cuda.synchronize() before starting and stopping the timer.
Otherwise the timing will be accumulated in the next blocking operation and will yield a wrong result.