Answers.Body: <p>Technically, nothing prevents you from doing so. When you have mulitple losses, you may call <code>.backward()</code> at each term separately.</p>
  <p>However, I wonder, whether it makes sense to optimize each individual path as a separate objective, since if we have multiple of them - we would like to solve several tasks simultaneously.</p>
  <p>Probably, it could be beneficial as some kind of regularization. One makes steps away from the gradient, but overall in the direction, which makes the model less prone to overfitting. But the choice of the <strong>batch size</strong>, learning rate - seem more straightforward way to achieve this. There is also a ~2x times additional computational overhead since we backpropagate twice.</p>
  <p>In some sense, It is done in the training of GAN's. Instead of backpropagating for discriminator and generator simultaneously - one calculates the loss separately for each model and updates weights.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-13T19:54:13.627Z
Answers.LastActivityDate: 2021-10-13T19:54:13.627Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.Body: <p>I am not sure if the process defined in the question is meaningful at all. If you mean to simply add the contribution of each <span class="math-container">$L$</span> without running the algorithm for the mini-batch, it makes no difference at all if you make one update or more; as the loss functions' contribution are simply added in the update. If on the other hand you mean to run the algorithm on the mini-batch once for each <span class="math-container">$L$</span> before calculating its contribution including the constraints, it would most probably end up with a step in the wrong direction. Constraints (at least <span class="math-container">$L_1$</span> and <span class="math-container">$L_2$</span>) do not depend on the input or the output of the model, they do not compare the prediction with the target value; they are functions of the weights only. There is no reason to expect trying to minimize this function by itself will improve the model; it will actually harm the progress by taking a (small or large) step in the wrong direction.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-13T11:21:13.477Z
Answers.Text: So, $k$ updates for each mini-batch is totally wrong?
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-13T11:30:11.107Z
Answers.Text: @hanugm It is k updates for each batch, where batch means the entire dataset. One update for each mini-batch.
Answers.UserDisplayName: serali
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-13T11:38:50.333Z
Answers.Text: Oh. In my question, I used the word "batch" instead of "mini-batch".
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-13T12:55:50.173Z
Answers.Text: Sorry to inform you that we made an edit. Please read it again if possible,
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-13T09:27:35.530Z
Answers.LastActivityDate: 2021-10-15T07:08:20.370Z
Answers.LastEditDate: 2021-10-15T07:08:20.370Z
Answers.UserDisplayName: serali
Body: <p>I am training a neural network using a mini-batch gradient descent algorithm.</p>
  <p>Now, consider the following loss function, which is composed of 2 terms.</p>
  <p><span class="math-container">$$L = L_{\text{MSE}} + L_{\text{regularization}} \label{1}\tag{1}$$</span></p>
  <p>As far as I understand, usually, we update the weights of a neural network only once per mini-batch, even if the loss function is composed of 2 or more terms, like in equation \ref{1}. So, in this approach, you calculate the 2 terms, add them, and then update weights once based on the sum.</p>
  <p>My question is: rather than summing the 2 terms of the loss function <span class="math-container">$L$</span> in equation \ref{1} and computing a single gradient for <span class="math-container">$L$</span>, couldn't we separately compute the gradient both for <span class="math-container">$L_{\text{MSE}}$</span> <span class="math-container">$L_{\text{regularization}}$</span>, then update the weights of the neural network twice? So, in this case, we would update the weights twice for each mini-batch. When would this make sense? Of course, my question also applies to the case where <span class="math-container">$L$</span> is composed of more than 2 terms.</p>
  
Tags: <deep-learning><objective-functions><mini-batch-gradient-descent>
Title: When would it make sense to perform a gradient descent step for each term of a loss function with multiple terms?
UserDisplayName: hanugm
ThreadText: # When would it make sense to perform a gradient descent step for each term of a loss function with multiple terms?
  
  (Asked by: username_0 on 2021-10-13)
  
  I am training a neural network using a mini-batch gradient descent algorithm.
  
  Now, consider the following loss function, which is composed of 2 terms.
  
  $$L = L\_{\text{MSE}} + L\_{\text{regularization}} \label{1}\tag{1}$$
  
  As far as I understand, usually, we update the weights of a neural network only once per mini-batch, even if the loss function is composed of 2 or more terms, like in equation \ref{1}. So, in this approach, you calculate the 2 terms, add them, and then update weights once based on the sum.
  
  My question is: rather than summing the 2 terms of the loss function $L$ in equation \ref{1} and computing a single gradient for $L$, couldn't we separately compute the gradient both for $L\_{\text{MSE}}$ $L\_{\text{regularization}}$, then update the weights of the neural network twice? So, in this case, we would update the weights twice for each mini-batch. When would this make sense? Of course, my question also applies to the case where $L$ is composed of more than 2 terms.
  
  **Tags:** deep-learning, objective-functions, mini-batch-gradient-descent
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-10-13. Score: 1
  
  Technically, nothing prevents you from doing so. When you have mulitple losses, you may call `.backward()` at each term separately.
  
  However, I wonder, whether it makes sense to optimize each individual path as a separate objective, since if we have multiple of them - we would like to solve several tasks simultaneously.
  
  Probably, it could be beneficial as some kind of regularization. One makes steps away from the gradient, but overall in the direction, which makes the model less prone to overfitting. But the choice of the **batch size**, learning rate - seem more straightforward way to achieve this. There is also a ~2x times additional computational overhead since we backpropagate twice.
  
  In some sense, It is done in the training of GAN's. Instead of backpropagating for discriminator and generator simultaneously - one calculates the loss separately for each model and updates weights.
  
  
  ---
  
  ### Answer by username_2 on 2021-10-13. Score: 1
  
  I am not sure if the process defined in the question is meaningful at all. If you mean to simply add the contribution of each $L$ without running the algorithm for the mini-batch, it makes no difference at all if you make one update or more; as the loss functions' contribution are simply added in the update. If on the other hand you mean to run the algorithm on the mini-batch once for each $L$ before calculating its contribution including the constraints, it would most probably end up with a step in the wrong direction. Constraints (at least $L\_1$ and $L\_2$) do not depend on the input or the output of the model, they do not compare the prediction with the target value; they are functions of the weights only. There is no reason to expect trying to minimize this function by itself will improve the model; it will actually harm the progress by taking a (small or large) step in the wrong direction.
  
  > **Comments:**
  > **username_0**: So, $k$ updates for each mini-batch is totally wrong?  
  > **username_2**: @hanugm It is k updates for each batch, where batch means the entire dataset. One update for each mini-batch.  
  > **username_0**: Oh. In my question, I used the word "batch" instead of "mini-batch".  
  > **username_0**: Sorry to inform you that we made an edit. Please read it again if possible,  
  
  ---
  
  