<p>For your use case, you want to update the model with a single <code>.backward()</code> call to a combined loss, similar to what you're doing in version 3. However, version 3 only calls the gradient update after going through both dataloaders - you want to do it every batch. Something like this</p>
<pre><code>ic_iter = iter(self._ic_loader)
res_iter = iter(self._res_loader)
for i in range(iterations):
    ic_input = next(ic_iter)
    u_hat = self._model(torch.cat(ic_input, dim=1))
    ic_loss = self._ic_loss(u_hat, ic_input)

    res_input = next(res_iter)
    u_hat = self._model(torch.cat(res_input, dim=1))
    res_loss = self._res_loss(u_hat, res_input)

    loss = ic_loss + alpha*res_loss # alpha scaling parameter
    loss.backward()
    self._optimizer.step()
    self._optimizer.zero_grad()
</code></pre>
<p>If the dataloaders are different lengths, you'll have to figure out the right cadence for refreshing the iterators.</p>

CC BY-SA 4.0
2023-10-17T19:46:26.000Z
Thanks to your reply that made me remember that I code on Python, I found an even simpler solution to this by making use of `sum()` function with embedding the "for-loop" as generators for both cases and summing them up for `backward()` as such: `ic_loss = sum(self._ic_loss(self._model(torch.cat(ic_input, dim=1)), ic_input) for ic_input in self._ic_loader)` with `(ic_loss + res_loss).backward()`.
Burak Karaosmanoğlu
CC BY-SA 4.0
2023-10-17T15:05:32.930Z
2023-10-17T15:05:32.930Z
Karl
<p>I am working on a model on PyTorch where it has two loss functions each fed from two separate input datasets. I want to update the model parameters based on these loss functions, <strong>ic_loss</strong>, <strong>res_loss</strong>, and also record their respective values each epoch. I highlighted by <em>### 1 ###</em>, <em>### 2 ###</em> and <em>### 3 ###</em> the lines of code that I modified. It seems to me that both <strong>Code 1)</strong> and <strong>Code 2)</strong> do the same thing, but I doubt otherwise. Here are three alternatives from the training part of my code that I am not sure of which to pick:</p>
<p><strong>Code 1)</strong></p>
<pre><code>    self._model.train()

    for epoch in range(1, self._epoch_number + 1):

      loss_info_string = f'Epoch number: [{epoch} / {self._epoch_number}] - '
      ic_loss_val = 0.0
      res_loss_val = 0.0

      for ic_input in self._ic_loader:

        u_hat = self._model(torch.cat(ic_input, dim=1))
        ic_loss = self._ic_loss(u_hat, ic_input)
        ic_loss.backward() ### 1 ###
        ic_loss_val += ic_loss.item()

      ic_loss_val /= len(self._ic_loader)

      loss_info_string += f'Initial Condition error/loss: {ic_loss_val:.6e}'

      for res_input in self._res_loader:

        u_hat = self._model(torch.cat(res_input, dim=1))
        res_loss = self._res_loss(u_hat, res_input)
        res_loss.backward() ### 1 ###
        res_loss_val += res_loss.item()

      res_loss_val /= len(self._res_loader)

      loss_info_string += f' - Residual error/loss: {res_loss:.6e}'

      self._optimizer.step() ### 2 ###
      self._optimizer.zero_grad() ### 3 ###

      loss_info_string += f' - Total error/loss: {ic_loss_val + res_loss_val:.6e}'
</code></pre>
<p><strong>Code 2)</strong></p>
<pre><code>    self._model.train()

    for epoch in range(1, self._epoch_number + 1):

      loss_info_string = f'Epoch number: [{epoch} / {self._epoch_number}] - '
      ic_loss_val = 0.0
      res_loss_val = 0.0

      for ic_input in self._ic_loader:

        u_hat = self._model(torch.cat(ic_input, dim=1))
        ic_loss = self._ic_loss(u_hat, ic_input)
        ic_loss.backward() ### 1 ###
        self._optimizer.step() ### 2 ###
        self._optimizer.zero_grad() ### 3 ###
        ic_loss_val += ic_loss.item()

      ic_loss_val /= len(self._ic_loader)

      loss_info_string += f'Initial Condition error/loss: {ic_loss_val:.6e}'

      for res_input in self._res_loader:

        u_hat = self._model(torch.cat(res_input, dim=1))
        res_loss = self._res_loss(u_hat, res_input)
        res_loss.backward() ### 1 ###
        self._optimizer.step() ### 2 ###
        self._optimizer.zero_grad() ### 3 ###
        res_loss_val += res_loss.item()

      res_loss_val /= len(self._res_loader)

      loss_info_string += f' - Residual error/loss: {res_loss:.6e}'

      loss_info_string += f' - Total error/loss: {ic_loss_val + res_loss_val:.6e}'
</code></pre>
<p><strong>Code 3)</strong></p>
<pre><code>    self._model.train()

    for epoch in range(1, self._epoch_number + 1):

      loss_info_string = f'Epoch number: [{epoch} / {self._epoch_number}] - '
      ic_loss_val = 0.0
      res_loss_val = 0.0

      for ic_input in self._ic_loader:

        u_hat = self._model(torch.cat(ic_input, dim=1))
        ic_loss = self._ic_loss(u_hat, ic_input)
        ic_loss_val += ic_loss.item()

      ic_loss_val /= len(self._ic_loader)

      loss_info_string += f'Initial Condition error/loss: {ic_loss_val:.6e}'

      for res_input in self._res_loader:

        u_hat = self._model(torch.cat(res_input, dim=1))
        res_loss = self._res_loss(u_hat, res_input)
        res_loss_val += res_loss.item()

      res_loss_val /= len(self._res_loader)

      loss_info_string += f' - Residual error/loss: {res_loss:.6e}'
      
      (ic_loss + res_loss).backward() ### I ###
      self._optimizer.step() ### II ###
      self._optimizer.zero_grad() ### III ###

      loss_info_string += f' - Total error/loss: {ic_loss_val + res_loss_val:.6e}'
</code></pre>
<p><strong>Code 1)</strong> and <strong>Code 3)</strong> seem to be the same except for if I used a batch size different from the entire datasets since <strong>Code 1)</strong> would accumulate gradients for each batch whereas <strong>Code 3)</strong> would use only the last batch. Therefore, I discarded <strong>Code 3)</strong>. Is that correct?</p>
<p>Also based on what I tried to described above, which one should I consider using, <strong>Code 1)</strong> or <strong>Code 2)</strong>? I hope everything is clear. Thanks in advance!</p>

<deep-learning><pytorch><loss>
Multiple Loss Functions For Proper Parameter Updates
Burak Karaosmanoğlu
# Multiple Loss Functions For Proper Parameter Updates

(Asked by: username_0 on 2023-10-17)

I am working on a model on PyTorch where it has two loss functions each fed from two separate input datasets. I want to update the model parameters based on these loss functions, ic_loss, res_loss, and also record their respective values each epoch. I highlighted by ### 1 ###, ### 2 ### and ### 3 ### the lines of code that I modified. It seems to me that both Code 1) and Code 2) do the same thing, but I doubt otherwise. Here are three alternatives from the training part of my code that I am not sure of which to pick:
Code 1)
    self._model.train()

    for epoch in range(1, self._epoch_number + 1):

      loss_info_string = f'Epoch number: [{epoch} / {self._epoch_number}] - '
      ic_loss_val = 0.0
      res_loss_val = 0.0

      for ic_input in self._ic_loader:

        u_hat = self._model(torch.cat(ic_input, dim=1))
        ic_loss = self._ic_loss(u_hat, ic_input)
        ic_loss.backward() ### 1 ###
        ic_loss_val += ic_loss.item()

      ic_loss_val /= len(self._ic_loader)

      loss_info_string += f'Initial Condition error/loss: {ic_loss_val:.6e}'

      for res_input in self._res_loader:

        u_hat = self._model(torch.cat(res_input, dim=1))
        res_loss = self._res_loss(u_hat, res_input)
        res_loss.backward() ### 1 ###
        res_loss_val += res_loss.item()

      res_loss_val /= len(self._res_loader)

      loss_info_string += f' - Residual error/loss: {res_loss:.6e}'

      self._optimizer.step() ### 2 ###
      self._optimizer.zero_grad() ### 3 ###

      loss_info_string += f' - Total error/loss: {ic_loss_val + res_loss_val:.6e}'

Code 2)
    self._model.train()

    for epoch in range(1, self._epoch_number + 1):

      loss_info_string = f'Epoch number: [{epoch} / {self._epoch_number}] - '
      ic_loss_val = 0.0
      res_loss_val = 0.0

      for ic_input in self._ic_loader:

        u_hat = self._model(torch.cat(ic_input, dim=1))
        ic_loss = self._ic_loss(u_hat, ic_input)
        ic_loss.backward() ### 1 ###
        self._optimizer.step() ### 2 ###
        self._optimizer.zero_grad() ### 3 ###
        ic_loss_val += ic_loss.item()

      ic_loss_val /= len(self._ic_loader)

      loss_info_string += f'Initial Condition error/loss: {ic_loss_val:.6e}'

      for res_input in self._res_loader:

        u_hat = self._model(torch.cat(res_input, dim=1))
        res_loss = self._res_loss(u_hat, res_input)
        res_loss.backward() ### 1 ###
        self._optimizer.step() ### 2 ###
        self._optimizer.zero_grad() ### 3 ###
        res_loss_val += res_loss.item()

      res_loss_val /= len(self._res_loader)

      loss_info_string += f' - Residual error/loss: {res_loss:.6e}'

      loss_info_string += f' - Total error/loss: {ic_loss_val + res_loss_val:.6e}'

Code 3)
    self._model.train()

    for epoch in range(1, self._epoch_number + 1):

      loss_info_string = f'Epoch number: [{epoch} / {self._epoch_number}] - '
      ic_loss_val = 0.0
      res_loss_val = 0.0

      for ic_input in self._ic_loader:

        u_hat = self._model(torch.cat(ic_input, dim=1))
        ic_loss = self._ic_loss(u_hat, ic_input)
        ic_loss_val += ic_loss.item()

      ic_loss_val /= len(self._ic_loader)

      loss_info_string += f'Initial Condition error/loss: {ic_loss_val:.6e}'

      for res_input in self._res_loader:

        u_hat = self._model(torch.cat(res_input, dim=1))
        res_loss = self._res_loss(u_hat, res_input)
        res_loss_val += res_loss.item()

      res_loss_val /= len(self._res_loader)

      loss_info_string += f' - Residual error/loss: {res_loss:.6e}'
      
      (ic_loss + res_loss).backward() ### I ###
      self._optimizer.step() ### II ###
      self._optimizer.zero_grad() ### III ###

      loss_info_string += f' - Total error/loss: {ic_loss_val + res_loss_val:.6e}'

Code 1) and Code 3) seem to be the same except for if I used a batch size different from the entire datasets since Code 1) would accumulate gradients for each batch whereas Code 3) would use only the last batch. Therefore, I discarded Code 3). Is that correct?
Also based on what I tried to described above, which one should I consider using, Code 1) or Code 2)? I hope everything is clear. Thanks in advance!

**Tags:** deep-learning, pytorch, loss


---

## 1 answers

### Answer by username_1 on 2023-10-17. Score: 1

For your use case, you want to update the model with a single .backward() call to a combined loss, similar to what you're doing in version 3. However, version 3 only calls the gradient update after going through both dataloaders - you want to do it every batch. Something like this
ic_iter = iter(self._ic_loader)
res_iter = iter(self._res_loader)
for i in range(iterations):
    ic_input = next(ic_iter)
    u_hat = self._model(torch.cat(ic_input, dim=1))
    ic_loss = self._ic_loss(u_hat, ic_input)

    res_input = next(res_iter)
    u_hat = self._model(torch.cat(res_input, dim=1))
    res_loss = self._res_loss(u_hat, res_input)

    loss = ic_loss + alpha*res_loss # alpha scaling parameter
    loss.backward()
    self._optimizer.step()
    self._optimizer.zero_grad()

If the dataloaders are different lengths, you'll have to figure out the right cadence for refreshing the iterators.

> **Comments:**
> **username_0**: Thanks to your reply that made me remember that I code on Python, I found an even simpler solution to this by making use of `sum()` function with embedding the "for-loop" as generators for both cases and summing them up for `backward()` as such: `ic_loss = sum(self._ic_loss(self._model(torch.cat(ic_input, dim=1)), ic_input) for ic_input in self._ic_loader)` with `(ic_loss + res_loss).backward()`.  

---

