Solved by tom in post #6 
              
              
                You need it before the forward pass. 
So technically, it’s not a Module but a Parameter/Tensor property and you need to decide before you use the Tensor whether you want to differentiate later so that the autograd can start recording. 
Best regards 
Thomas