alwynmathew:

How to dynamically add modules to a pre-exsisting model?


You can use model.add_module, but would also need to change the forward method or use this new module manually.



 alwynmathew:

How to update optimizer (state_dict) without losing old info?


optimizer.add_param_group can be used to add new parameters.



 alwynmathew:

How can I use initially nn.Identity in my model and later replace it with Conv2d after N steps?


This might work, but I would recommend to double check, if this layer is really used (e.g. via forward hooks) and make sure the parameters are added to the optimizer.