Solved by googlebot in post #10 
              
              
                when some Function needs to keep an input tensor, it is stored locally in saved_tensors - this is the same for built-in operators, they just use c++ version of Function. There is an obvious disconnect between Module and Function, plus some gradients are computed without storing function inputs. 
For…