Answers.Body: <p>For your use case, you want to update the model with a single <code>.backward()</code> call to a combined loss, similar to what you're doing in version 3. However, version 3 only calls the gradient update after going through both dataloaders - you want to do it every batch. Something like this</p>
  <pre><code>ic_iter = iter(self._ic_loader)
  res_iter = iter(self._res_loader)
  for i in range(iterations):
      ic_input = next(ic_iter)
      u_hat = self._model(torch.cat(ic_input, dim=1))
      ic_loss = self._ic_loss(u_hat, ic_input)
  
      res_input = next(res_iter)
      u_hat = self._model(torch.cat(res_input, dim=1))
      res_loss = self._res_loss(u_hat, res_input)
  
      loss = ic_loss + alpha*res_loss # alpha scaling parameter
      loss.backward()
      self._optimizer.step()
      self._optimizer.zero_grad()
  </code></pre>
  <p>If the dataloaders are different lengths, you'll have to figure out the right cadence for refreshing the iterators.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-17T19:46:26.000Z
Answers.Text: Thanks to your reply that made me remember that I code on Python, I found an even simpler solution to this by making use of `sum()` function with embedding the "for-loop" as generators for both cases and summing them up for `backward()` as such: `ic_loss = sum(self._ic_loss(self._model(torch.cat(ic_input, dim=1)), ic_input) for ic_input in self._ic_loader)` with `(ic_loss + res_loss).backward()`.
Answers.UserDisplayName: Burak Karaosmanoğlu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-17T15:05:32.930Z
Answers.LastActivityDate: 2023-10-17T15:05:32.930Z
Answers.UserDisplayName: Karl
Body: <p>I am working on a model on PyTorch where it has two loss functions each fed from two separate input datasets. I want to update the model parameters based on these loss functions, <strong>ic_loss</strong>, <strong>res_loss</strong>, and also record their respective values each epoch. I highlighted by <em>### 1 ###</em>, <em>### 2 ###</em> and <em>### 3 ###</em> the lines of code that I modified. It seems to me that both <strong>Code 1)</strong> and <strong>Code 2)</strong> do the same thing, but I doubt otherwise. Here are three alternatives from the training part of my code that I am not sure of which to pick:</p>
  <p><strong>Code 1)</strong></p>
  <pre><code>    self._model.train()
  
      for epoch in range(1, self._epoch_number + 1):
  
        loss_info_string = f'Epoch number: [{epoch} / {self._epoch_number}] - '
        ic_loss_val = 0.0
        res_loss_val = 0.0
  
        for ic_input in self._ic_loader:
  
          u_hat = self._model(torch.cat(ic_input, dim=1))
          ic_loss = self._ic_loss(u_hat, ic_input)
          ic_loss.backward() ### 1 ###
          ic_loss_val += ic_loss.item()
  
        ic_loss_val /= len(self._ic_loader)
  
        loss_info_string += f'Initial Condition error/loss: {ic_loss_val:.6e}'
  
        for res_input in self._res_loader:
  
          u_hat = self._model(torch.cat(res_input, dim=1))
          res_loss = self._res_loss(u_hat, res_input)
          res_loss.backward() ### 1 ###
          res_loss_val += res_loss.item()
  
        res_loss_val /= len(self._res_loader)
  
        loss_info_string += f' - Residual error/loss: {res_loss:.6e}'
  
        self._optimizer.step() ### 2 ###
        self._optimizer.zero_grad() ### 3 ###
  
        loss_info_string += f' - Total error/loss: {ic_loss_val + res_loss_val:.6e}'
  </code></pre>
  <p><strong>Code 2)</strong></p>
  <pre><code>    self._model.train()
  
      for epoch in range(1, self._epoch_number + 1):
  
        loss_info_string = f'Epoch number: [{epoch} / {self._epoch_number}] - '
        ic_loss_val = 0.0
        res_loss_val = 0.0
  
        for ic_input in self._ic_loader:
  
          u_hat = self._model(torch.cat(ic_input, dim=1))
          ic_loss = self._ic_loss(u_hat, ic_input)
          ic_loss.backward() ### 1 ###
          self._optimizer.step() ### 2 ###
          self._optimizer.zero_grad() ### 3 ###
          ic_loss_val += ic_loss.item()
  
        ic_loss_val /= len(self._ic_loader)
  
        loss_info_string += f'Initial Condition error/loss: {ic_loss_val:.6e}'
  
        for res_input in self._res_loader:
  
          u_hat = self._model(torch.cat(res_input, dim=1))
          res_loss = self._res_loss(u_hat, res_input)
          res_loss.backward() ### 1 ###
          self._optimizer.step() ### 2 ###
          self._optimizer.zero_grad() ### 3 ###
          res_loss_val += res_loss.item()
  
        res_loss_val /= len(self._res_loader)
  
        loss_info_string += f' - Residual error/loss: {res_loss:.6e}'
  
        loss_info_string += f' - Total error/loss: {ic_loss_val + res_loss_val:.6e}'
  </code></pre>
  <p><strong>Code 3)</strong></p>
  <pre><code>    self._model.train()
  
      for epoch in range(1, self._epoch_number + 1):
  
        loss_info_string = f'Epoch number: [{epoch} / {self._epoch_number}] - '
        ic_loss_val = 0.0
        res_loss_val = 0.0
  
        for ic_input in self._ic_loader:
  
          u_hat = self._model(torch.cat(ic_input, dim=1))
          ic_loss = self._ic_loss(u_hat, ic_input)
          ic_loss_val += ic_loss.item()
  
        ic_loss_val /= len(self._ic_loader)
  
        loss_info_string += f'Initial Condition error/loss: {ic_loss_val:.6e}'
  
        for res_input in self._res_loader:
  
          u_hat = self._model(torch.cat(res_input, dim=1))
          res_loss = self._res_loss(u_hat, res_input)
          res_loss_val += res_loss.item()
  
        res_loss_val /= len(self._res_loader)
  
        loss_info_string += f' - Residual error/loss: {res_loss:.6e}'
        
        (ic_loss + res_loss).backward() ### I ###
        self._optimizer.step() ### II ###
        self._optimizer.zero_grad() ### III ###
  
        loss_info_string += f' - Total error/loss: {ic_loss_val + res_loss_val:.6e}'
  </code></pre>
  <p><strong>Code 1)</strong> and <strong>Code 3)</strong> seem to be the same except for if I used a batch size different from the entire datasets since <strong>Code 1)</strong> would accumulate gradients for each batch whereas <strong>Code 3)</strong> would use only the last batch. Therefore, I discarded <strong>Code 3)</strong>. Is that correct?</p>
  <p>Also based on what I tried to described above, which one should I consider using, <strong>Code 1)</strong> or <strong>Code 2)</strong>? I hope everything is clear. Thanks in advance!</p>
  
Tags: <deep-learning><pytorch><loss>
Title: Multiple Loss Functions For Proper Parameter Updates
UserDisplayName: Burak Karaosmanoğlu
ThreadText: # Multiple Loss Functions For Proper Parameter Updates
  
  (Asked by: username_0 on 2023-10-17)
  
  I am working on a model on PyTorch where it has two loss functions each fed from two separate input datasets. I want to update the model parameters based on these loss functions, **ic\_loss**, **res\_loss**, and also record their respective values each epoch. I highlighted by *### 1 ###*, *### 2 ###* and *### 3 ###* the lines of code that I modified. It seems to me that both **Code 1)** and **Code 2)** do the same thing, but I doubt otherwise. Here are three alternatives from the training part of my code that I am not sure of which to pick:
  
  **Code 1)**
  
  ```
      self._model.train()
  
      for epoch in range(1, self._epoch_number + 1):
  
        loss_info_string = f'Epoch number: [{epoch} / {self._epoch_number}] - '
        ic_loss_val = 0.0
        res_loss_val = 0.0
  
        for ic_input in self._ic_loader:
  
          u_hat = self._model(torch.cat(ic_input, dim=1))
          ic_loss = self._ic_loss(u_hat, ic_input)
          ic_loss.backward() ### 1 ###
          ic_loss_val += ic_loss.item()
  
        ic_loss_val /= len(self._ic_loader)
  
        loss_info_string += f'Initial Condition error/loss: {ic_loss_val:.6e}'
  
        for res_input in self._res_loader:
  
          u_hat = self._model(torch.cat(res_input, dim=1))
          res_loss = self._res_loss(u_hat, res_input)
          res_loss.backward() ### 1 ###
          res_loss_val += res_loss.item()
  
        res_loss_val /= len(self._res_loader)
  
        loss_info_string += f' - Residual error/loss: {res_loss:.6e}'
  
        self._optimizer.step() ### 2 ###
        self._optimizer.zero_grad() ### 3 ###
  
        loss_info_string += f' - Total error/loss: {ic_loss_val + res_loss_val:.6e}'
  
  ```
  
  **Code 2)**
  
  ```
      self._model.train()
  
      for epoch in range(1, self._epoch_number + 1):
  
        loss_info_string = f'Epoch number: [{epoch} / {self._epoch_number}] - '
        ic_loss_val = 0.0
        res_loss_val = 0.0
  
        for ic_input in self._ic_loader:
  
          u_hat = self._model(torch.cat(ic_input, dim=1))
          ic_loss = self._ic_loss(u_hat, ic_input)
          ic_loss.backward() ### 1 ###
          self._optimizer.step() ### 2 ###
          self._optimizer.zero_grad() ### 3 ###
          ic_loss_val += ic_loss.item()
  
        ic_loss_val /= len(self._ic_loader)
  
        loss_info_string += f'Initial Condition error/loss: {ic_loss_val:.6e}'
  
        for res_input in self._res_loader:
  
          u_hat = self._model(torch.cat(res_input, dim=1))
          res_loss = self._res_loss(u_hat, res_input)
          res_loss.backward() ### 1 ###
          self._optimizer.step() ### 2 ###
          self._optimizer.zero_grad() ### 3 ###
          res_loss_val += res_loss.item()
  
        res_loss_val /= len(self._res_loader)
  
        loss_info_string += f' - Residual error/loss: {res_loss:.6e}'
  
        loss_info_string += f' - Total error/loss: {ic_loss_val + res_loss_val:.6e}'
  
  ```
  
  **Code 3)**
  
  ```
      self._model.train()
  
      for epoch in range(1, self._epoch_number + 1):
  
        loss_info_string = f'Epoch number: [{epoch} / {self._epoch_number}] - '
        ic_loss_val = 0.0
        res_loss_val = 0.0
  
        for ic_input in self._ic_loader:
  
          u_hat = self._model(torch.cat(ic_input, dim=1))
          ic_loss = self._ic_loss(u_hat, ic_input)
          ic_loss_val += ic_loss.item()
  
        ic_loss_val /= len(self._ic_loader)
  
        loss_info_string += f'Initial Condition error/loss: {ic_loss_val:.6e}'
  
        for res_input in self._res_loader:
  
          u_hat = self._model(torch.cat(res_input, dim=1))
          res_loss = self._res_loss(u_hat, res_input)
          res_loss_val += res_loss.item()
  
        res_loss_val /= len(self._res_loader)
  
        loss_info_string += f' - Residual error/loss: {res_loss:.6e}'
        
        (ic_loss + res_loss).backward() ### I ###
        self._optimizer.step() ### II ###
        self._optimizer.zero_grad() ### III ###
  
        loss_info_string += f' - Total error/loss: {ic_loss_val + res_loss_val:.6e}'
  
  ```
  
  **Code 1)** and **Code 3)** seem to be the same except for if I used a batch size different from the entire datasets since **Code 1)** would accumulate gradients for each batch whereas **Code 3)** would use only the last batch. Therefore, I discarded **Code 3)**. Is that correct?
  
  Also based on what I tried to described above, which one should I consider using, **Code 1)** or **Code 2)**? I hope everything is clear. Thanks in advance!
  
  **Tags:** deep-learning, pytorch, loss
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-10-17. Score: 1
  
  For your use case, you want to update the model with a single `.backward()` call to a combined loss, similar to what you're doing in version 3. However, version 3 only calls the gradient update after going through both dataloaders - you want to do it every batch. Something like this
  
  ```
  ic_iter = iter(self._ic_loader)
  res_iter = iter(self._res_loader)
  for i in range(iterations):
      ic_input = next(ic_iter)
      u_hat = self._model(torch.cat(ic_input, dim=1))
      ic_loss = self._ic_loss(u_hat, ic_input)
  
      res_input = next(res_iter)
      u_hat = self._model(torch.cat(res_input, dim=1))
      res_loss = self._res_loss(u_hat, res_input)
  
      loss = ic_loss + alpha*res_loss # alpha scaling parameter
      loss.backward()
      self._optimizer.step()
      self._optimizer.zero_grad()
  
  ```
  
  If the dataloaders are different lengths, you'll have to figure out the right cadence for refreshing the iterators.
  
  > **Comments:**
  > **username_0**: Thanks to your reply that made me remember that I code on Python, I found an even simpler solution to this by making use of `sum()` function with embedding the "for-loop" as generators for both cases and summing them up for `backward()` as such: `ic_loss = sum(self._ic_loss(self._model(torch.cat(ic_input, dim=1)), ic_input) for ic_input in self._ic_loader)` with `(ic_loss + res_loss).backward()`.  
  
  ---
  
  