Solved by ludwigwinkler in post #2 
              
              
                I got a version running that satisfies my needs. 
The trick is to pass the module itself in the forward function such that it can be called by reference. 
The main issue was that although I could compute the per sample gradients during the backward call, there was no reference to the parent module s…