mattmcc97:

#Freeze decoder for p in model2.parameters(): p.requires_grad = False


I think you are almost there. Place this code before using model2. i.e., at the top after load_state_dict.
(or)
Do not specify the model2 parameters to the optimizer. i.e., only specify FCN parameters.