Adam has a separate learning rate for each parameter. The param_group['lr'] is a kind of base learning rate that does not change. There is no variable in the PyTorch Adam implementation that stores the dynamic learning rates.
One could save the optimizer state, as mentioned here:

  
    
    
    Saving and loading a model in Pytorch? 
  
  
    If I have a model class and a trainer class. I create an instance of the model and train it. 
model = mymodel()
train = trainer.train(model...) 

How can I save the model to a file, after it has been trained and how can I then reload it and continue training? I searched for this but didn’t get an answer.
  


The PyTorch implementation of Adam can be found here:
https://pytorch.org/docs/stable/_modules/torch/optim/adam.html 310
The line for p in group['params']: iterates over all the parameters and calculates the learning rate for each parameter.
I found this helpful: http://ruder.io/optimizing-gradient-descent/index.html#adam 417