ptrblck:

…the CPU seems to run ahead with the data loading (assuming “Import data…” is used in the dataset) while the GPU is executing the workload…


Yes, the CPU part is in my dataset class (torch.utils.data.Dataset) … That is in def __get_item__(self, idx), I have to download numpy data from .h5 file (I used it for the smaller file size)  and convert them into pytorch tensor and cuda device.
So, could you further confirm how should I decide which part is the runtime of my network (inference)…? I have marked it as Net @ [iter 57] 14.693 ms with NVTX…
However, the reason for me to be quite confused is that …in the previously attached Nsight profiler, the (below) light blue range measured at CUDA is much longer than the (above) runtime measured at NVTX (Net @ [iter 57] 14.693 ms ).  So, was it 14.693 ms? (But when I measured with Torch.cuda.event, the runtime was much longer (73ms)…? )
Or the longer range at CUDA than NVTX in the Nsight Profiler does not mean that it takes longer? Did I miss measuring anything or doing something wrong?
I am so sorry. I am quite new in understanding the Profiler.  Please help.