@jkf - If we do not do zero_grad, then what will happen is that the the gradients of all your previous batches will keep adding up for a particular weight say w and the weight will get updated with the sum of all the previous gradients instead of the gradient of the current batch.