So pytorch’s main strength is the ability to have dynamic computation graphs. I was wondering if there is a simple way of initializing or inserting new layers into the network as well while the network is training? For example, epoch 0 to 10 trains a single linear layer (input -> 100) and then epoch 10 to 20 adds in another layer (100 -> 1000) with random weights. Thus layer 1 is being finetuned whereas layer 2 starts training from scratch. I realize one way could be to initialize all layers one might expect to be inserted and then dynamically change the data flow. However, I wanted to dynamically initialize new layers as well. Any related documentation/code?