Solved by ptrblck in post #4 
              
              
                Thanks for the debugging. 
As long as you recreate the optimizer, it should work. 
You could check the optimizer.state_dict()['state'] for the running estimates. If they are not empty, something is wrong. 
Otherwise, I agree that the result is looking unexpected, but the data might be the next thing…