Do you use the same cudnn version in both cases?
Also make sure to run your code a few times before doing the timing. cudnn benchmarking will run the forward multiple times to find the best kernel if it is enabled.
What is the input size for the batchnorm? Does running just the batchnorm with this input size gives similar behavior?