Hi,
This is technically possible by setting the weights to the pretrained layers yourself with something like model.layer1.load_state_dict(pretrained_layer1_state_dict).
That being said, the insertion of your custom layers might change the behavior of the network completely and make the initialization useless.