So, without looking at all the details or being knowledgeable in the domain (I do nlp), the logic of loading, freezing and finetuning looks good to me. I have done the same before for relation prediction and it worked for me: save a pretrained model, load it, freeze the layers and add new layers and then finetune.
For what it’s worth, I haven’t been able to successfully improve performance by pretraining either.
What exactly is your data situation? When I pretrained with distant supervision data and finetuned with manually annotated data, my performance was about 30-40% worse than the regular model trained on manually annotated data.
Edit: Are you certain that this doesn’t create copies and that you freeze the respective layers? Have you printed requires_grad after this point to check?
        self.features = nn.Sequential(*list(netOrig.children())[:1])
        for p in self.features.parameters():
          p.requires_grad = False