I have implemented a custom conv layer, much like here 2. However it is still memory inefficient compared to native pytorch convlution layer, and slower. How can I fix both speed and memory consumption issues? Any drastic measure is accepted, like coding in the lowest level and building from that, since these issues are really important to me. An ideal situation is where my conv layer is as fast and memory efficient as the native one. I would appreciate any suggestions.