I have a NN with 4 heads that I want to train one after the other.
After having trained 1 head, I save the net and I then want to resume the model, change the requires_grad of the already trained head and start training the next head but I get this error:
ValueError: loaded state dict contains a parameter group that doesn’t match the size of optimizer’s group
I get the same error even if  the requires_grad is changed after having loaded the model.
I do not get any error if I do not make any change to requires_grad. Meaning that resume works fine.
The problem arises when I load the optimiser. The model can be loaded without problems.
Observe that the the Net structure never changes in time. It is the trainability of some layers that are switched.
How can I solve this problem?