Solved by braindotai in post #2 
              
              
                optimizer.grad_zero() resets gradients of parameters to zero. By not calling it, this means when you call .backward() to your loss, then this will add the current gradients (wrt current loss) to the gradients that were already assigned to the parameters (wrt previous loss). 
So here basically you’re…