Dynamically adding new params to optimizer seems to be harder that I though in pytorch. Even though, I found a solution to do it, here 2. This post author itself doesn’t recommend using it.

However, here’s the reason why you should probably never update your optimizer like this, but should instead re-initialize from scratch, and just accept the loss of state information


So - to sum it up: I’d really recommend to try to keep it simple, and to only change a parameter as conservatively as possible, and not to touch the optimizer.

I tried re-initialize optimizer from scratch whenever a new module is added to the network and I saw huge drop in validation which I cant sacrifice. Is there a better way to solve this problem?
cc: @ptrblck