You could save the optimizer state dict. Something like this suggested here: https://github.com/pytorch/pytorch/issues/2830#issuecomment-336194949 2.1k should work.
After that you could just load it back.
I am guessing the reason you see the same lr is that the lr for adam has not changed. The effective lr which has components from moment estimate are different.