I assume you’re synchronized the code to get a proper profiling result, if you are running on the GPU.
Anyway, the most likely reason is, that some convolution setups are highly optimized while others not so much.