Answers.Body: <p>Accumulating the loss like that doesn't improve the memory requirements, because the memory consumption depends on the size of your computational graph. In other words, each time you add a term to the loss, the overall function of your loss grows as well, and with it grows the memory consumption.</p>
  <p>Specifically, every time you add a term to the loss value, you run the forward step with a new input/output pair <span class="math-container">$x_i; y_i$</span>. In order to backpropagate, all the hidden activations <span class="math-container">$h_{i,l}$</span> must be stored until you call <code>loss.backward()</code>. So with each new loss added, that cache of latent activation vectors grows. Check out <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#computational-graph" rel="nofollow noreferrer">PyTorchs walkthrough</a> of what exactly happens there.</p>
  <p>In contrast, when you have already computed the gradient, then the cached activation vectors can be deleted and the only things that are saved are the gradient values for your model parameters. In comparison, accumulating the loss would lead to a memory consumption of <span class="math-container">$\mathcal{O}(n * I)$</span> and accumulating gradients is <span class="math-container">$\mathcal{O}(n + |\theta|) \sim \mathcal{O}(n)$</span>, where <span class="math-container">$n$</span> is your single-pass memory requirement, <span class="math-container">$I$</span> is the number of accumulation steps, and <span class="math-container">$|\theta|$</span> is the number of model parameters. I hope that was somehow clear.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-30T09:58:29.310Z
Answers.Text: So in short, loss accumulation does not achieve the effect of reduce memory requirement. And in the case of multi-task training, what is your opinion if I change loss accumulation to gradient accumulation?
Answers.UserDisplayName: LSM
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-30T10:46:08.520Z
Answers.Text: It is mathematically equivalent whether you backpropagate a loss $L(x, y) = l_1(x, y) + l_2(x, y)$, or you accumulate the gradients from separate backpropagation of $l_1$ and $l_2$. So if you can save memory using gradient accumulation I'd say go for it :)
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-27T11:22:05.117Z
Answers.LastActivityDate: 2022-11-27T11:29:40.103Z
Answers.LastEditDate: 2022-11-27T11:29:40.103Z
Answers.UserDisplayName: Chillston
Body: <p>I have read and used Gradient Accumulation as a method to handle large batch size on smaller memory restrictions. It is described as following:</p>
  <pre><code>for step, eachBatch in enumerate(dataloader):
     ... 
     loss = loss_func(ytrue, ypred)
     loss.backward()
     if step % 5 ==0: 
        # update weights every 5 steps
        loss.step()
        loss.zero_grad()
  </code></pre>
  <p>However, instead of accumulating the gradient, can we accumulate loss values? Just like multi-task trainings, but I have never seen people recommending the following method, but wonder why it that the case, what are the potential problems with this. If there really exist a problem with the following method, then why does loss accumulation work for multi-task training?</p>
  <pre><code>totalLoss = 0
  for step, eachBatch in enumerate(dataloader):
     ... 
     loss = loss_func(ytrue, ypred)
     totalLoss += loss # accumulate the loss
     if step % 5 ==0: 
        totalLoss.backward()
        totalLoss.step()
        totalLoss.zero_grad()
        totalLoss=0
  </code></pre>
  
Tags: <neural-networks><deep-learning><computational-complexity><multi-task-learning><space-complexity>
Title: Instead of accumulating the gradient, can we accumulate loss values?
UserDisplayName: LSM
ThreadText: # Instead of accumulating the gradient, can we accumulate loss values?
  
  (Asked by: username_0 on 2022-11-27)
  
  I have read and used Gradient Accumulation as a method to handle large batch size on smaller memory restrictions. It is described as following:
  for step, eachBatch in enumerate(dataloader):
     ... 
     loss = loss_func(ytrue, ypred)
     loss.backward()
     if step % 5 ==0: 
        # update weights every 5 steps
        loss.step()
        loss.zero_grad()
  
  However, instead of accumulating the gradient, can we accumulate loss values? Just like multi-task trainings, but I have never seen people recommending the following method, but wonder why it that the case, what are the potential problems with this. If there really exist a problem with the following method, then why does loss accumulation work for multi-task training?
  totalLoss = 0
  for step, eachBatch in enumerate(dataloader):
     ... 
     loss = loss_func(ytrue, ypred)
     totalLoss += loss # accumulate the loss
     if step % 5 ==0: 
        totalLoss.backward()
        totalLoss.step()
        totalLoss.zero_grad()
        totalLoss=0
  
  **Tags:** neural-networks, deep-learning, computational-complexity, multi-task-learning, space-complexity
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-11-27. Score: 2
  
  Accumulating the loss like that doesn't improve the memory requirements, because the memory consumption depends on the size of your computational graph. In other words, each time you add a term to the loss, the overall function of your loss grows as well, and with it grows the memory consumption.
  
  Specifically, every time you add a term to the loss value, you run the forward step with a new input/output pair $x\_i; y\_i$. In order to backpropagate, all the hidden activations $h\_{i,l}$ must be stored until you call `loss.backward()`. So with each new loss added, that cache of latent activation vectors grows. Check out [PyTorchs walkthrough](https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#computational-graph) of what exactly happens there.
  
  In contrast, when you have already computed the gradient, then the cached activation vectors can be deleted and the only things that are saved are the gradient values for your model parameters. In comparison, accumulating the loss would lead to a memory consumption of $\mathcal{O}(n \* I)$ and accumulating gradients is $\mathcal{O}(n + |\theta|) \sim \mathcal{O}(n)$, where $n$ is your single-pass memory requirement, $I$ is the number of accumulation steps, and $|\theta|$ is the number of model parameters. I hope that was somehow clear.
  
  > **Comments:**
  > **username_0**: So in short, loss accumulation does not achieve the effect of reduce memory requirement. And in the case of multi-task training, what is your opinion if I change loss accumulation to gradient accumulation?  
  > **username_1**: It is mathematically equivalent whether you backpropagate a loss $L(x, y) = l_1(x, y) + l_2(x, y)$, or you accumulate the gradients from separate backpropagation of $l_1$ and $l_2$. So if you can save memory using gradient accumulation I'd say go for it :)  
  
  ---
  
  