Thanks for reply, the missing of the last synchronization is a typo, the actual script is corret, sorry for that mistake.
When change torch.backends.cudnn.benchmark to False, the inference time are the same for all batches. However, another problem occurs, the inference time of all batches get much slower, because there are dilated convolutions in my model. Accoding to this issue https://github.com/pytorch/pytorch/issues/15054 21, set cudnn as True can solve it, so these two problems are contradictory. Any idea to solve it ?
Thanks a lot.