@ptrblck I’ve loaded the pretrained weights, set the requires_grad attribute of the pretrained layers to False and then proceeded to train the main model. I called Model.state_dict() on the trained main model and visualised the weights but it appears that the weights of the pretrained layers were altered.
Shown below are the first 5 weights of one of the pretrained layers before and after training the main model.
Before training:
-6.1031e-02, -4.0205e-02, -8.9633e-02, -1.9241e-02, -1.2121e-02
After training:
-6.5355e-02, -4.4506e-02, -9.4377e-02, -2.1566e-02, -1.1395e-02
It seems that setting the requires_grad attribute of the pretrained layers to False did not work as intended. Should I be using Optimiser = torch.nn.Adam(Model.parameters()) or Optimiser = torch.nn.Adam(Model.(Layer to be trained).parameters())?