Yes, in that case you shouldn’t zero out the gradients in each iteration and note that each backward() call would accumulate the gradients. In batch training the gradients are usually calculated using the mean, so you might also need to scale the gradients before applying the optimizer.step() operation.