I suppose that it may be the design choice. The designer set a buffer to keep the accumulated sum of every parameter’s grad. For every training step, loss.backward() calculate the current grads for every parameter, and then dump them to the buffer. After that, we can use optimizer.step() to apply these grads which stored on the buffer to update parameters. So if we don’t use w1.grad.data.zero_() or optimizer.zero_grad() before update the parameters, we will apply the accumulated grads to update them, which may be wrong in most situations.