Solved by ptrblck in post #3 
              
              
                Yes. A loop does not detach the computation graph and as long as you are using PyTorch methods, Autograd will create the backward pass for you. 
If you need to leave PyTorch and e.g. use numpy in the forward pass, you would need to implement the backward function manually.