I’m a bit confused about the numbers.
Did you make sure to synchronize the device before starting and stopping the timer (or did you profile the kernels via nsight)?
For cudnn benchmark, some warm-up iterations are necessary in addition as @albanD explained.
What is your use case by the way?
If you want to apply soma operation on each small kernel, you might want to unfold the image, and perform the op e.g. via matmul.