DenisDiachkov:

Is there any way to make it in pytorch


Yes, you could start by implementing the desired work flow using e.g. nested for loops and once it’s working use unfold to apply a matmul to speed it up, if that fits your use case.



 DenisDiachkov:

and keep it as efficient as regular convolution?


That won’t be straightforward, but you could e.g. write a custom CUDA extension for your convolution to get another speedup.