Answers.Body: <p>I am currently working on a similar problem. I think your approach is good. As for setting the parameter lambda, since you are using deep neural networks, you can make it a learnable parameter, instead of a hyperparameter you set. This way, as the two losses fluctuate over your training iterations/epochs, the model will be able to adjust the lambda parameter accordingly.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-14T13:42:31.930Z
Answers.Text: This will not work as a parameter. The network will just set the harder task to zero. Must be a hyper param
Answers.UserDisplayName: chessprogrammer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-19T12:17:16.703Z
Answers.Text: You can add (1/lambda) to penalize your model for making the weight zero.
Answers.UserDisplayName: carbocation
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-25T19:22:00.210Z
Answers.LastActivityDate: 2021-11-25T19:22:00.210Z
Answers.UserDisplayName: rachkov91
Body: <p>Say I'm training a model for multiple tasks by trying to minimize sum of losses <span class="math-container">$L_1 + L_2$</span> via gradient descent.</p>
  <p>If these losses are on a different scale, the one whose range is greater will dominate the optimization. I'm currently trying to fix this problem by introducing a hyperparameter <span class="math-container">$\lambda$</span>, and trying to bring these losses to the same scale by tuning it, i.e., I try to minimize <span class="math-container">$L_1 +\lambda \cdot L_2$</span> where <span class="math-container">$\lambda &gt; 0 $</span>.</p>
  <p>However, I'm not sure if this is a good approach. In short, what are some strategies to deal with losses having different scales when doing multi-task learning? I'm particularly interested in deep learning scenarios.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-18T13:18:32.823Z
Comments.Text: I've seen multiple times the approach that you're currently using in different contexts, such as Bayesian neural networks and multi-objective optimization with evolutionary algorithms, so your approach is at least not new and may work for your purposes. Off the top of my head, no other completely different approach comes to my mind, but it's possible there are others. So, I think the main question/problem is not whether your approach is good or not, but how to scale the losses. Maybe this Wikipedia article: https://en.wikipedia.org/wiki/Multi-objective_optimization can be useful.
Comments.UserDisplayName: nbro
Tags: <deep-learning><objective-functions><gradient-descent><multi-task-learning>
Title: How to deal with losses on different scales in multi-task learning?
UserDisplayName: SpiderRico
ThreadText: # How to deal with losses on different scales in multi-task learning?
  
  (Asked by: username_0 on 2021-03-18)
  
  Say I'm training a model for multiple tasks by trying to minimize sum of losses $L\_1 + L\_2$ via gradient descent.
  
  If these losses are on a different scale, the one whose range is greater will dominate the optimization. I'm currently trying to fix this problem by introducing a hyperparameter $\lambda$, and trying to bring these losses to the same scale by tuning it, i.e., I try to minimize $L\_1 +\lambda \cdot L\_2$ where $\lambda > 0 $.
  
  However, I'm not sure if this is a good approach. In short, what are some strategies to deal with losses having different scales when doing multi-task learning? I'm particularly interested in deep learning scenarios.
  
  **Tags:** deep-learning, objective-functions, gradient-descent, multi-task-learning
  > **Comments:**
  > **username_1**: I've seen multiple times the approach that you're currently using in different contexts, such as Bayesian neural networks and multi-objective optimization with evolutionary algorithms, so your approach is at least not new and may work for your purposes. Off the top of my head, no other completely different approach comes to my mind, but it's possible there are others. So, I think the main question/problem is not whether your approach is good or not, but how to scale the losses. Maybe this Wikipedia article: https://en.wikipedia.org/wiki/Multi-objective_optimization can be useful.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-11-25. Score: 1
  
  I am currently working on a similar problem. I think your approach is good. As for setting the parameter lambda, since you are using deep neural networks, you can make it a learnable parameter, instead of a hyperparameter you set. This way, as the two losses fluctuate over your training iterations/epochs, the model will be able to adjust the lambda parameter accordingly.
  
  > **Comments:**
  > **username_3**: This will not work as a parameter. The network will just set the harder task to zero. Must be a hyper param  
  > **username_4**: You can add (1/lambda) to penalize your model for making the weight zero.  
  
  ---
  
  