Solved by Varal7 in post #5 
              
              
                Under the hood, yes, the autograd engine does compute the gradient of the loss with respect to the output but the result is never stored but passed along backward through the graph to compute the gradient with respect to the leaves. 
What are leaves: that’s exactly the tensors created with requires_…