I found that cudnn configuration has a great influence on the average inference time. So I tried different cudnn configuration strategy. Drawn from the experiment:

cudnn.benchmark=True or cudnn.deterministic=True can improve the inference time, but it is randomly.
when I set them both False the average inference time is more stable, the upper and lower gap is small around 1fps, but it is slower than the first condition.
If there has another task run on the same GPU with you, it also can influence the testing of inference time.

I am not sure if I was right or not, does anyone have ideas on it?
EDIT:

It seems that we should pass some random data to the network firstly, because it needs a “warm up”.