Do you have a small code sample I can use to experiment locally? Because I’m not sure what the 3.5s means here. (Input size / batch size / cuda,cudnn version / hardware etc)
One  common cuda perf pitfalls is when we end up launching many many kernels which do very small tasks. So you spend more time launching the kernel than doing compute.

This can be because of the internal implementation (THCUNN or cudnn) that does some assumptions (small batch size here) that you don’t.
This can be because your code does these ops and it should be batched differently.

For the first point, there is no magic kernel to perform all convolutions, so many specialized kernels are needed to get the best perf. It’s possible that such specialized kernel is not in cudnn