@Dwight_Foster As I mentioned, I want the model to treat both losses equally i.e. both should go down ideally. So, I don’t want to discard custom loss at some point of time.
@rahulvigneswaran
First point is not possible, custom loss can be negative. I already tried second one, but then custom loss stopped decreasing i.e. over entire training I saw almost flat line of loss, while cross entropy was indeed decreasing. It made things even worse. So, I guess it was focusing on just cross entropy loss.
I was thinking to implement a dynamic multiplier instead of static which will be multiplied with custom loss i.e. this multiplier will also change over entire training.
Any idea if that makes sense, how to decide which equation (e.g. ax**2+b…) should that multiplier follow and how it can be implemented in training loop?
Thank you for the reply.