Hi!  I would like to ask more questions about Nsight profiling vs using torch.cuda.Event…
The problem that I have is that the runtime of Nsight profiling is much faster (about 14 ms) than using torch.cuda.Event (about 74 ms)…
My questions are


What are the differences between the two? Is it because the Nsight profiling is running on Asynchronization mode, but the using torch.cuda.Event is running on Synchronization mode.


Which of these two results should I use as the reference for the runtime performance of my network?


Details on how I ran Nsight Profiler and torch.cuda.event are as follows:


Nsight Profiler. I have used the Nsight Profiler with command line

nsys profile -t cuda,osrt,nvtx,cudnn,cublas -o profile_inference_runtime -w true  -f true python3 demo_inference_runtime.py

which gives the following runtime results. Notice that the (below)  light blue range measured at CUDA is much longer than the (above) runtime measured at NVTX (also denoted as Net @ [iter 57] 14.693 ms):

Screenshot from 2021-05-02 22-10-591920×1080 274 KB



Torch.cuda.event, I measured the runtime with torch.cuda.Event as follows.  :

torch.cuda.synchronize()
start_net_event = torch.cuda.Event(enable_timing=True)
end_net_event = torch.cuda.Event(enable_timing=True)
start_net_event.record()

out_t = net(pred_data) 

end_net_event.record()
torch.cuda.synchronize()   
net_runtime_temp = start_net_event.elapsed_time(end_net_event) 

which gives the following runtime results.

Screenshot from 2021-05-02 22-20-241318×192 38.6 KB