My insight was that loading the state dict after defining the optimizer was making the optimizer to update “other” weights (the ones corresponding to the model before replacing them by the loaded ones).
If that’s not the case you will probably have some bug anywhere else.