Solved by ptrblck in post #2 
              
              
                No, each backward call accumulates the gradients in the .grad attribute of the used parameters, which is also why you have to call optimizer/model.zero_grad() before calculating the gradients of a new iteration.