Answers.Body: <p>This is an important subfield within multi-task learning, called gradient combination. Here is a list of about a dozen recent approaches: <a href="https://github.com/Manchery/awesome-multi-task-learning#loss--gradient-strategy" rel="nofollow noreferrer">https://github.com/Manchery/awesome-multi-task-learning#loss--gradient-strategy</a></p>
  <p>In particular, this paper is a good starting place: <a href="https://arxiv.org/abs/2001.06782" rel="nofollow noreferrer">https://arxiv.org/abs/2001.06782</a></p>
  <p>A brief summary of the work done on this topic:
  It is often better to analytically combine the gradients, than to directly combine the losses and then get one gradient. Many ways have been proposed to do this, some involving gradient norms, some involving dynamic rescaling of gradients, some involving geometric analysis, and some involving game theory.</p>
  <p>I should also note: there is an entirely different approach to multi objective optimization called Pareto Optimality, where instead of simply combining the losses in one particular way, we analyze a set of different ways of combining the objectives and their trade-offs. For more info, this is a good starting point: <a href="https://en.wikipedia.org/wiki/Pareto_front" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Pareto_front</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-24T19:09:57.900Z
Answers.LastActivityDate: 2023-05-24T20:21:43.867Z
Answers.LastEditDate: 2023-05-24T20:21:43.867Z
Answers.UserDisplayName: chessprogrammer
Body: <p>I am optimizing a neural network with Adam using 3 different losses. Their scale is very different, and the current method is to either sum the losses and clip the gradient or to manually weight them within the sum. Something like: <span class="math-container">$clip(w_1\nabla_{L_1} + w_2\nabla_{L_2} + w_3\nabla_{L_3}, c)$</span>.</p>
  <p>I am thinking of better approaches. My current idea is to clip gradients separately (to avoid having one gradient &quot;overtaking&quot; the others too much), then weigh them, sum them, and finally clip them (with a smaller threshold than the one used for the first clipping).
  Something like: <span class="math-container">$clip(w_1clip(\nabla_{L_1},c) + w_2clip(\nabla_{L_2},c) + w_3clip(\nabla_{L_3},c), c_2)$</span>.</p>
  <p>I am not sure what the best way to weigh them would be, though. Like having weights <span class="math-container">$w_i$</span> proportional to their gradient norm?
  I'd like to get some suggestions / references.</p>
  
Tags: <objective-functions><gradient-descent><regularization><gradient-clipping>
Title: What is the best way to combine or weight multiple losses with gradient descent?
UserDisplayName: Simon
ThreadText: # What is the best way to combine or weight multiple losses with gradient descent?
  
  (Asked by: username_0 on 2023-05-24)
  
  I am optimizing a neural network with Adam using 3 different losses. Their scale is very different, and the current method is to either sum the losses and clip the gradient or to manually weight them within the sum. Something like: $clip(w\_1\nabla\_{L\_1} + w\_2\nabla\_{L\_2} + w\_3\nabla\_{L\_3}, c)$.
  
  I am thinking of better approaches. My current idea is to clip gradients separately (to avoid having one gradient "overtaking" the others too much), then weigh them, sum them, and finally clip them (with a smaller threshold than the one used for the first clipping).
  Something like: $clip(w\_1clip(\nabla\_{L\_1},c) + w\_2clip(\nabla\_{L\_2},c) + w\_3clip(\nabla\_{L\_3},c), c\_2)$.
  
  I am not sure what the best way to weigh them would be, though. Like having weights $w\_i$ proportional to their gradient norm?
  I'd like to get some suggestions / references.
  
  **Tags:** objective-functions, gradient-descent, regularization, gradient-clipping
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-05-24. Score: 6 | Accepted answer
  
  This is an important subfield within multi-task learning, called gradient combination. Here is a list of about a dozen recent approaches: <https://github.com/Manchery/awesome-multi-task-learning#loss--gradient-strategy>
  
  In particular, this paper is a good starting place: <https://arxiv.org/abs/2001.06782>
  
  A brief summary of the work done on this topic:
  It is often better to analytically combine the gradients, than to directly combine the losses and then get one gradient. Many ways have been proposed to do this, some involving gradient norms, some involving dynamic rescaling of gradients, some involving geometric analysis, and some involving game theory.
  
  I should also note: there is an entirely different approach to multi objective optimization called Pareto Optimality, where instead of simply combining the losses in one particular way, we analyze a set of different ways of combining the objectives and their trade-offs. For more info, this is a good starting point: <https://en.wikipedia.org/wiki/Pareto_front>
  
  
  ---
  
  