Solved by ptrblck in post #2 
              
              
                I’m not sure you will gain that much from filtering out the other layers, as usually you just initialize the model once before the training. 
Would it be possible to just initialize all layers and load the pretrained parameters after it or is your use case different? 
If this would work, here is a s…