optimizer.grad_zero() resets gradients of parameters to zero. By not calling it, this means when you call .backward() to your loss, then this will add the current gradients (wrt current loss) to the gradients that were already assigned to the parameters (wrt previous loss).
So here basically you’re updating your model parameters with large gradients (since gradients are adding as we move on to next batch).
Now in case of SGD, which is a simple algorithm without any complications, this might work as this algorithm is already pretty slow, so in this case not calling optimizer.zero_grad() is providing you large gradients, hence SGD might work better in this case. Also my guess is your model is also quite simple (unlike resnet20).
But if you’re model is large and you’re using other gradient descent varients like adam, nadam, rmsprop… etc, then this approach will give nan values.