Answers.Body: <ol>
  <li>not generally, consider a case where <span class="math-container">$L_2 = 1/L_1$</span>, if one converges to 0, the other one diverges</li>
  <li>Yes, that's the usual definition of convergence, though the &quot;<span class="math-container">$\approx$</span>&quot; is not really well defined usually</li>
  <li>exactly, however, after one update, most likely you will have that <span class="math-container">$\frac{dL_1}{d\theta}$</span> won't be 0 anymore</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-23T09:42:01.797Z
Answers.LastActivityDate: 2024-01-23T09:42:01.797Z
Answers.UserDisplayName: Alberto
Body: <p>I am training a deep learning model, the loss function of which is of the form</p>
  <p><span class="math-container">$$
  \cal{L} = \cal{L_1} + \cal{L_2}
  $$</span></p>
  <p>where <span class="math-container">$\cal{L_1}$</span> and <span class="math-container">$\cal{L_2}$</span> are of very different orders. WLOG, let's assume the order of <span class="math-container">$\cal{L_1}$</span> is much higher than the order of <span class="math-container">$\cal{L_2}$</span>.</p>
  <p>During the first several epochs of training, the model will attempt to minimize <span class="math-container">$\cal{L_1}$</span> largely. However, after a certain number of epochs, the value of <span class="math-container">$\cal{L_1}$</span> will converge.</p>
  <p>My question is, what will happen now? Specifically, I have three questions:</p>
  <ul>
  <li><p>Does the convergence of <span class="math-container">$\cal{L_1}$</span> imply the convergence of <span class="math-container">$\cal{L}$</span>, which means the training is over and the loss function behaved as if it was essentially <span class="math-container">$\cal{L} = \cal{L_1}$</span>?</p>
  </li>
  <li><p>Since <span class="math-container">$\cal{L_1}$</span> has now converged, does that imply <span class="math-container">$\frac{\partial{\cal{L_1}}}{\partial{\theta}} \approx 0$</span>? (where <span class="math-container">$\theta$</span> are the model parameters)</p>
  </li>
  <li><p>If the above point is true, then since the model parameters are updated based on <span class="math-container">$\frac{\partial{\cal{L}}}{\partial \theta}$</span>,  does that imply that the model will now start minimizing <span class="math-container">$\cal{L_2}$</span> (since <span class="math-container">$\frac{\partial{\cal{L}}}{\partial \theta} \approx \frac{\partial \cal{L_2}}{\partial \theta}$</span>)?</p>
  </li>
  </ul>
  
Tags: <neural-networks><machine-learning><deep-learning>
Title: Impact of scaling in loss terms when loss function is a composition of multiple functions
UserDisplayName: user185887
ThreadText: # Impact of scaling in loss terms when loss function is a composition of multiple functions
  
  (Asked by: username_0 on 2024-01-22)
  
  I am training a deep learning model, the loss function of which is of the form
  
  $$
  \cal{L} = \cal{L\_1} + \cal{L\_2}
  $$
  
  where $\cal{L\_1}$ and $\cal{L\_2}$ are of very different orders. WLOG, let's assume the order of $\cal{L\_1}$ is much higher than the order of $\cal{L\_2}$.
  
  During the first several epochs of training, the model will attempt to minimize $\cal{L\_1}$ largely. However, after a certain number of epochs, the value of $\cal{L\_1}$ will converge.
  
  My question is, what will happen now? Specifically, I have three questions:
  
  * Does the convergence of $\cal{L\_1}$ imply the convergence of $\cal{L}$, which means the training is over and the loss function behaved as if it was essentially $\cal{L} = \cal{L\_1}$?
  * Since $\cal{L\_1}$ has now converged, does that imply $\frac{\partial{\cal{L\_1}}}{\partial{\theta}} \approx 0$? (where $\theta$ are the model parameters)
  * If the above point is true, then since the model parameters are updated based on $\frac{\partial{\cal{L}}}{\partial \theta}$, does that imply that the model will now start minimizing $\cal{L\_2}$ (since $\frac{\partial{\cal{L}}}{\partial \theta} \approx \frac{\partial \cal{L\_2}}{\partial \theta}$)?
  
  **Tags:** neural-networks, machine-learning, deep-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-23. Score: 0 | Accepted answer
  
  1. not generally, consider a case where $L\_2 = 1/L\_1$, if one converges to 0, the other one diverges
  2. Yes, that's the usual definition of convergence, though the "$\approx$" is not really well defined usually
  3. exactly, however, after one update, most likely you will have that $\frac{dL\_1}{d\theta}$ won't be 0 anymore
  
  
  ---
  
  