But do you actually run out of memory?
It is expected that pytorch will allocate more and more memory for the first iterations then it will stabilize after a while.
If you do run out of memory, you should try and pinpoint where it happens. Try replacing your cpp code by a python version the just returns an output of the right size. And see if it still happens.