Thank you for your fast reply and thorough explanation.
I am now using cuda events for benchmarking on a NVIDIA Xavier in 30W mode and I am getting lower latency values than the ones in the paper and also a different value for each run ! I measure the model.evaluate() part of the code so I am not sure if I am on the right track.