You can check whether CuDNN is used using e.g. nvprof to list the kernels that are being run. This might also give you insights what else is going on.
Many people asking about performance oddities forget to synchronize the GPU before the measurement begins and ends, this may or may not be the case in your testing.
Best regards
Thomas