Yes, CUDA operations are executed asynchronously so to profile them you would need to synchronize the code before starting and stopping the timers via torch.cuda.synchronize().



 taehyunzzz:

From observations, it seems that the total time for a kernel cannot be calculated simply as “cpu_time + cuda_time”.


I don’t understand this claim. If you are not synchronizing, the CPU would run ahead and just stop the timer directly while the GPU is still busy.