Thanks for the answer. I am not sure what you meant my synchronizing the code before the GPU run. Can you please elaborate? I just timed the inference time on a single sample as follows:
net=Model().to(device).eval()
t1=time.time()
flow=net(frames)
t2=time.time()
print("Inference time =", t2-t1)

There is no data loading in the timed part of the code as frames is a preloaded tensor. I also tried adding torch.backends.cudnn.benchmark = True as you suggested, but it does not result in any significant change in the inference time in either case.