anyxp:

However , this hardcoded approach does not scale. In the end I would like to insert non-trivial layers dynamically into a several models at several positions.
How do I achieve this?


My opinion would be initialize the model then copy the weights back except the new layers. By this way you can be pretty sure all the things are connected in the network.
You can refer to any dynamically expanding network  implementation for this.