Solved by ptrblck in post #10 
              
              
                If you need to have more flexibility, you could have a look at forward hooks in this post. 
Since you need the gradients, make sure not to detach the tensor.