CUDA operations are executed asynchronously so to you would need to synchronize the code via torch.cuda.synchronize() before starting and stopping the timers.
Otherwise you would profile the kernel launches, will see accumulated times in blocking operations etc., which seems to be the case here.