I’m trying to dynamically keep adding new modules to my network after N steps as in Progressive Growing of GAN 1.
But I came across this info from @Carl about PyTorch from here:

Dynamically adding/removing modules is relatively easy in Tensorflow/Keras, since a graph of the model is available on the python side. In PyTorch, you cannot traverse the graph of your model to insert modules.

@Carl also mention that we can make:

layers act like the identity function

and train the network.
Questions:

How to dynamically add modules to a pre-exsisting model?
How to update optimizer (state_dict) without losing old info?
How can I use initially nn.Identity in my model and later replace it with Conv2d after N steps?

Any help is appreciated. Thank you.