I can send a PR, just wanted to first discuss how to structure the code.
I keep detaching, cause for my specific needs didn’t need to keep track of these variables in autograd and wanted to explicitly free up the memory (or at least that was the intention).