CUDA operations are executed asynchronously. If you don’t synchronize the code, blocking operations (such as the .cpu() op) could accumulate the execution time from previous CUDA calls.
To get a better profile I would recommend to take a look at the timeline via the PyTorch Profiler 1 or Nsight Systems.