georgepsh:

But anyway, so is the reasonable explanation a low-level implementations of the convolutions? I need to also note that I received such results on CPU, not GPU. Can it depend on the hardware it runs on?
Can it also depend on the framework?


Primarily it depends on kernel, but hardware can affect this too. For example MKL-DNN done by Intel, so one would expect better perfomance on Intel CPUs. You can use torch.profiler to estimate performance difference.