Hi everyone,
I’m profiling a large network with torch.autograd.profiler.profile(use_cuda=True). As far as I can tell, I run everything on the GPU, but still get quite a few operations where the profiler reports much more time spent on the CPU than on the GPU. Some examples are:
N5torch8autograd13CopyBackwardsE:
     CPU: 87555.573us      GPU: 1532.959us
mse_loss:
     CPU: 1767.147us      GPU: 93.445us
avg_pool3d_backward:
     CPU: 2655.118us      GPU: 183.594us
add:
     CPU: 10984.388us      GPU: 131.836us
CatBackward:
     CPU: 24874.143us      GPU: 13.672us

I’m expecting that some of this has to do with the lack of synchronization between CPU and GPU (when I call torch.cuda.synchronize(), some values change).
I’ve tried other profilers (nvprof+nvvp) and get similar results.
My question: If this issue is due to the lack of synchronization, how do I get more accurate execution times?
If it is not due to synchronization and the values are correct, are there tricks to reduce the amount of CPU time? I expected most of these functions to be highly parallelized and benefit greatly from GPU kernels…