Ah I see.
So theoretically it would be interesting to see what the “optimizer state” consists of and if the state is dependent on the parameters, which it tries to train.
If the optimizer does not depent on the parameters there will be no issue with loading the current state with different parameters.
If it is dependent, it is questionable if the dependence is separated for each parameter.
In this case it may be possible to load the state of the optimizer without all parameters.
In the case of the optimizer having the second derivative saved in its state, it will not be possible. Or at least I would not know how .