I forgot to mention that grouped convolutions in my example work faster than the regular ones. Both for decomposed and non-decomposed cases.
But anyway, so is the reasonable explanation a low-level implementations of the convolutions? I need to also note that I received such results on CPU, not GPU. Can it depend on the hardware it runs on?
Can it also depend on the framework?