Solved by TheShadow29 in post #2 
              
              
                You could save the optimizer state dict. Something like this suggested here: https://github.com/pytorch/pytorch/issues/2830#issuecomment-336194949 should work. 
After that you could just load it back. 
I am guessing the reason you see the same lr is that the lr for adam has not changed. The effectiv…