I’ve created a small comparison using nn.Conv2d and nn.Linear for a random input of [10, 64, 128, 128].
As I’ve used unfold and permute for the linear implementation, it’s slower on my machine compared to the conv approach. Maybe you have another idea how to avoid the (expensive) shape translations.
Here 74 is the gist.