i re-run the timing with cuda.event, and i get relatively the same time except the first sample when using python time i get way higher time than torch.cuda.event. i dont know why is that.
not sure what python time was computing? and why this huge time at the first sample? this happens constantly on every first sample of any minibatch.
what is the true time that my code will take (it cant be 2 different times)? is it the one computed with torch.cuda.event or python time? if python time is taking this long, someone must be doing something that torch.cuda.event didnt take in consideration… it could be something to do with synchronization. when serializing the execution, i get consistent time in both timers.
could torch.histc creates a synchronization point at the first sample allowing all the kernels over other samples to be done, therefore it is fast on the rest of samples but not the first one?
the goal of this measurements is to find bottlenecks to improve their speed and speedup the full code.
in this case, for every minibatch, 99% of time is spent on first sample everytime. it is fine if this happens only once. but this happens in every first sample of every minibatch. so, it costs.
thanks
here is all the timings:

time using time.perf_counter  (s)


with CUDA_LAUNCH_BLOCKING=1

sample 1 histc 0.0004032440483570099 
sample 2 histc 0.0005275830626487732  
sample 3 histc 0.000552598387002945 


without  CUDA_LAUNCH_BLOCKING=1

sample 1 histc 0.3368970938026905  **WHY?** 
sample 2 histc 0.00043926388025283813 
sample 3 histc 0.0003735348582267761 


time using torch.cuda.Event (ms)


with CUDA_LAUNCH_BLOCKING=1

sample 1 histc 0.4039680063724518
sample 2 histc 0.6740800142288208
sample 3 histc 5.139200210571289


without  CUDA_LAUNCH_BLOCKING=1

sample 1 histc 0.3036479949951172
sample 2 histc 0.47996801137924194
sample 3 histc 0.5324159860610962

general setup:
def function(tensor):
    # operations ...
    # tx = time.perf_counter()
    start_event = torch.cuda.Event(enable_timing=True)
    end_event = torch.cuda.Event(enable_timing=True)
    start_event.record()

    hist = torch.histc(x, bins=bins)
    
    end_event.record()
    elapsed_time_ms = start_event.elapsed_time(end_event)
    print('histc {}'.format(elapsed_time_ms))
    # print('histc {} '.format(time.perf_counter() - tx))
  
for epoch in epochs:
  forward(minibatch)
  # prepare tensors
  l_tensors = None
  for t in range(minibatch_size):
    l_tensors.vstack((l_tesnors, compute(tensor_t))
   
  # process all tensors.
  for tensor in l_tensors:
    function(tensor) # <-- where we measure time.