I am sorry I do not have test script now because the original code is really a large project.
I test the FPN-RPN network for detection task. The input size is not fixed, if I set torch.backends.cudnn.benchmark = True, the speed is even slower for it will seek algorithms every iteration which introduce extra overhead.
When I test FP32 bits, it is normal, cudnn.enabled will make the program faster; while if the input is FP16, cudnn.enabled will make it slower.