Hi Ali,
The loss variable is reassigned every iteration due to the line.
loss = criterion(output, input) / N

The momentum of the gradient is maintained by the optimizer for the parameters/tensors that it is tracking, in this case those are model.parameters().
Hope this helps.