Is this the very first execution of this layer in this script or are you seeing this increased time in another epoch of the model?
I would not recommend to profile the first execution for the aforementioned reasons (CUDA context creation, cudnnFind in benchmark mode, GPU startup/warmup time).