You could have searched…

  
    
    
    Pytorch loss does not decrease 
  
  
    You can save and load the optimizer state in a similar way to the model state. 
torch.save(optimizer.state_dict(), filename)
optimizer.load_state_dict(torch.load(filename))