The time spent on the device (and thus the performance) not only depends on the theoretical number of operations, but also on the implemented algorithm. E.g. if you are using cudnn, you might see a difference between highly optimized kernels for common use cases and “uncommon” cases.
I’m not sure, if you are using cudnn for your workloads and thus cannot comment on the applied kernels internally for your convs.