Solved by ptrblck in post #2 
              
              
                Yes, the computation graph will be created while running the specified operations (usually in your model’s forward, but it can be created by any arbitrary operation with a tensor which requires gradients). 
If you don’t specify retain_graph=True, the computation graph will be cleared after the backw…