If you are using a standard optimizer are you calling optimizer.zero_grad() to zero out the gradients before each step? Example: examples/main.py at cbb760d5e50a03df667cdc32a61f75ac28e11cbf · pytorch/examples · GitHub 2
Otherwise they will continue to accumulate with each update.