You call zero grad at the start of the mini-batch. If you do not your gradients accumulate. When you zero_grad only the gradients are zero’d out not the weights. Once you do a forward() and loss.backward() the gradients will be propagated. You can accumulate the gradients by not calling optimizer.zero_grad()