Solved by soulitzer in post #2 
              
              
                This is pretty tricky indeed, but this behavior is noted in the docs: 

By default, all the output Tensors that are of differentiable type will be set to require gradient and have all autograd metadata set for them. If you don’t want them to require gradients, you can use the mark_non_differentiable…