By default the gradients will be accumulated in the parameters.



 Ali_Akgoz:

When backpropagation finishes at the next step their weights will be different from each other.


That shouldn’t be the case, since the parameters are not updated during the backward() call, but by the optimizer.step().