I have been looking into the internals but its kind of tricky as the optimizer uses 'id’s, which change with each run, instead of parameters names as in the model state dict.
I guess for now the best/easiest solution is to make a new optimizer.
I have couple of follow-up questions about the optimizer I will ask in a separate post as they are more generic.