ptrblck:

n though are are currently only using a single Dataset ?
If so, then setting requires_grad=False for the linear layer,


Thank you for your response.
I am having a resnet appended with a linear FC layer for dataset 1, say FC-Dataset-1.
resnet = ...
in_fea = resnet.logits.in_features
resnet.logits = nn.Linear(in_features = in_fea, out_features = num_sub_1, bias = True)

In the meantime, there is another linear FC layer for dataset 2, say FC-Dataset-2
new_logits = nn.Linear(in_features = in_feas, out_features = num_sub_2, bias = True)

To train the new_logits only (while freezing FC-Dataset-1), I simply set the .requires_grad for FC-Dataset-1 to zero as follows.
for name, param in resnet.named_parameters():
        if name in ['logits.weight', 'logits.bias']:
            param.requires_grad = False
        else:
            param.requires_grad = True

And during training, I neglected the output for FC-Dataset-1 as follows.
# x, y_pred_1 = resnet(x, y)
x, _ = resnet(x, y) 
y_pred_2 = new_logits(x)

# loss_1 = loss_fn_1(y_pred_1, y)
loss_1 = 0
loss_2 = loss_fn_2(y_pred_2, y)
loss = loss_1 + loss_2

#  Involved only 1 optimizer combining parameters for both ends        
optimizer.zero_grad()  
loss.backward()
optimizer.step()

Is my implementation reasonable? thank you very much.