Hi thanks for replying
Actually my Network is different from I posted, which is Siamese Architecture and the loss function have 4 parts
Actually the loss was working correctly before adding the multiplication matrix, where its weights are not updated during training.
loss =  loss1 + loss2 + loss3+ 0.3*loss4
loss2 is calculated from the shap1 output using focal loss function