Solved by ptrblck in post #2 
              
              
                It depends on your use case. 
The usual approach would be to zero out the old gradients, calculate the new gradients via backward(), and update the parameters via optimizer.step() for each batch (so once per iteration). 
However, you could simulate a larger batch size by accumulating the gradients u…