Are you using a learning rate scheduler at the moment or are you manipulating the lr manually?
I would try to load the state_dict and set the learning rate to the desired value.
An uninitialized Adam optimizer might create loss spikes when the training continues, which might take some training again to lower it to the previous value.