It could be you filter out some parameters in the new optimizer, resulting mismatch. Removing the filter should resolve the issue.
Saving the state_dict of optimizer and scheduler helps to resume training.