did more digging.
didnt arrive to the exact cause of the problem, until now.
note, i have a network where i perform a forward for training over minibatch x, then i have the part of the code that processes z.
the network forward operates directly on minibatch  (parallel).
however, the part operating on z, works sequentially, only at the end, i gathered what batchable into single cuda operation over entire batch.
initially, i perform:
network(x)
process(z)

this causes the discussed issue when arriving to process(z) where first sample is very slow .33sec, then rest of samples are fast .003 with mintach times to .45sec.
then, i switched the order to
process(z)
network(x)

things works properly. problem ‘disappeared’ at process(z). all samples in z are processed in .003 and total z in .11sec without an explicit torch.cuda.synchronize().
but that’s not all.
network(x) is completely independent from process(z) in term of data or functions. so, i dont know how the order of executing kernels will affect them. also, this doesnt necessarily mean the problem is solved because the overall time of 1 epoch didnt improve much (from 2min14s to 2min03s, i expected more gain, like .33se * 188 minibatches). cuda is probably doing some implicit synch.
so, did some more digging. it turns out i was wrong about the time of network(x). it is not .001, but .33sec. i used the wrong tool to measure time cuda stuff using python time.
so, the extra .33sec were coming from the network forward and not other operations.
the order does not matter.
cuda needs to synch to get the model output causing the delay.
sorry for the inconvenience, and thanks for your help.