I have 4 losses which is used for backward out of which one is custom loss. These losses have different scaling. For example, Loss1 and Loss2 are in range of e-01, Loss3 is in range of e+01. Through hook i checked the gradients at output are in range of e-04 to e-05 for Loss1 to Loss3. Now custom Loss4 have high value and when it backpropagate to output it gives the gradients in range of e+03 to e+04.
First, i tried to scale Loss4 but there is no improvement in result. Second, I scale the gradient of Loss4 at output through hook but it doesn’t have any effect. I apply all possible scaling. I have used Adam optimizer.
Am i going in right direction or missing something? How to choose hyperparameter value for result improvement?