arman-yekkehkhani:

However it is still memory inefficient compared to native pytorch convlution layer, and slower.


You could directly reuse the native convolutions used in PyTorch to speed up your slower approach.
The cuDNN conv implementations are closed source, so you would need to check (open) reference implementations in CUDA (e.g. you could check the convs in cutlass as another reference).