Solved by robertknight in post #5 
              
              
                Hi Tom, 
The problem is that the accumulated gradients are not reset before each iteration. Therefore the optimizer is updating the weights based on not just the gradients from the current iteration but also all previous iterations. As explained in the torch.optim docs you need to call optimizer.zer…