Dear ptrblck,
Thanks for your reply,
Actually you are right the shared point is the output of (the replacement of self.res_model.fc)  nn.Sequential model (which is 1024 )
The idea here, I am using CosFace  self.classifier along side with another FC layer for classification,
In more details,
In the training phase,

Foreword pass: the two branches are used   self.classifier (Cosface) and self.res_model.classifier (the classification output), but the loss is calculated from just the classification branch,
back-propagation, the loss calculated from the classification output is used to back propagate, (no use of the cosface)

In the testing phase:
only the classification branch is active (the cosface was not used at all)
My main question, since I am not using the CosFace branch to calculate the loss and back propagate it, how it influence my training (recall the results are very different from when I calculate the loss from the CosFace and back propagate it, also when I dont use the CosFace at all)
Best Regards