Solved by ptrblck in post #2 
              
              
                Both approaches are valid for the standard use case, i.e. if you do not want to accumulate gradients for multiple iterations. 
You can thus call optimizer.zero_grad() everywhere in the loop but not between the loss.backward() and optimizer.step() operation.