Solved by sio277 in post #2 
              
              
                Yes. If you use the optimizer containing estimates of gradients, such as Adam and AdamW, also try to save and load its state_dict().