Solved by ptrblck in post #2 
              
              
                Based on your description I understand that you are calling optimizer.step() more often (1 out of 100 steps) and calculate the gradients only 1 out of 1000 steps. 
In this case, the general problem could be that the optimizer updates the parameters with “old” gradients, which might not work. 
To cha…