Well the benchmark you show seems to indicate that the batchnorm function is the problem. If you have cudnn enabled, then pytorch does nothing and just forwards to it.
What is the input size for the batchnorm? Does running just the batchnorm with this input size gives similar behavior?