Hello,
I have a model with 5 different losses as shown below:
cls_loss_function = HeatmapLoss(reduction='mean')  # Custom Loss Function
txty_loss_function = nn.BCEWithLogitsLoss(reduction='none')
twth_loss_function = nn.SmoothL1Loss(reduction='none')
iou_loss_function = nn.SmoothL1Loss(reduction='none')
dep_loss_funciton = nn.SmoothL1Loss(reduction='none')

The losses are calculated and then the total_loss is returned as shown below.
total_loss = cls_loss + txty_loss + twth_loss + iou_loss + dep_loss total_loss.backward()
However, txty_loss dominates the total_loss. For example: After 50 Epochs with batch_size 3, Adam with lr=1e-4, GroupNorm with frozen BN layers of ResNet18 which is used as an Encoder, all the other losses except txty_loss are 0, but txty_loss ranges between 10 to 17. I think the model gives bad predictions because it is dominated by txty_loss. Could this be right?
Note: I have tried using SGD and Adam with different learning rates but the behaviour remains the same.
If so, how do I add weights to these losses, as in: Assigning coefficients to each loss but keeping the sum of those coefficients 1 ?. The concept should be similar to the weighted BCE, I think.
Feel free to correct if the concepts are wrong.
Thank You.