Solved by tom in post #2 
              
              
                It’s likely not memory efficiency that matters here (you’ll have other ops that use more memory and you could do this after the optimizer step when memory is less precious than after the forward) but that you change a parameter that has you want inplace. 
The implementation you propose looks like t…