Answers.Body: <p>That is because the default reduction for the CEP loss calculation is mean. Hence to find the true average across all batches, you first multiply by the batch size and then divide by total number of data points. Note this only matters when all the batches are not equal, i.e. total number of data points do not perfectly divide batch size. Else you can simply add all the batch losses and divide by number of batches.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-16T16:47:10.363Z
Answers.LastActivityDate: 2022-11-16T16:47:10.363Z
Answers.UserDisplayName: Kushal Kumar
Body: <p>I am trying to look through a code of the transformer model from Pytorch. However,
  I do not understand why batch size needs to multiply with cross-entropy loss given that loss is calculated based on data at a given timestep.</p>
  <p>This is from the line: <strong>&quot;total_loss += batch_size * criterion(output_flat, targets).item()&quot;</strong></p>
  <p><a href="https://github.com/pytorch/tutorials/blob/master/beginner_source/transformer_tutorial.py#L323" rel="nofollow noreferrer">This is the section</a> of code:</p>
  <pre><code>def evaluate(model: nn.Module, eval_data: Tensor) -&gt; float:
      model.eval()  # turn on evaluation mode
      total_loss = 0.
      src_mask = generate_square_subsequent_mask(bptt).to(device)
      with torch.no_grad():
          for i in range(0, eval_data.size(0) - 1, bptt):
              data, targets = get_batch(eval_data, i)
              batch_size = data.size(0)
              if batch_size != bptt:
                  src_mask = src_mask[:batch_size, :batch_size]
              output = model(data, src_mask)
              output_flat = output.view(-1, ntokens)
              total_loss += batch_size * criterion(output_flat, targets).item()
      return total_loss / (len(eval_data) - 1)
  </code></pre>
  
Tags: <transformer><pytorch><cross-entropy><evaluation-functions>
Title: why cross entropy loss has to be multiplied by a batch size during an evaluation in transformer model?
UserDisplayName: BOB
ThreadText: # why cross entropy loss has to be multiplied by a batch size during an evaluation in transformer model?
  
  (Asked by: username_0 on 2022-07-19)
  
  I am trying to look through a code of the transformer model from Pytorch. However,
  I do not understand why batch size needs to multiply with cross-entropy loss given that loss is calculated based on data at a given timestep.
  
  This is from the line: **"total\_loss += batch\_size \* criterion(output\_flat, targets).item()"**
  
  [This is the section](https://github.com/pytorch/tutorials/blob/master/beginner_source/transformer_tutorial.py#L323) of code:
  
  ```
  def evaluate(model: nn.Module, eval_data: Tensor) -> float:
      model.eval()  # turn on evaluation mode
      total_loss = 0.
      src_mask = generate_square_subsequent_mask(bptt).to(device)
      with torch.no_grad():
          for i in range(0, eval_data.size(0) - 1, bptt):
              data, targets = get_batch(eval_data, i)
              batch_size = data.size(0)
              if batch_size != bptt:
                  src_mask = src_mask[:batch_size, :batch_size]
              output = model(data, src_mask)
              output_flat = output.view(-1, ntokens)
              total_loss += batch_size * criterion(output_flat, targets).item()
      return total_loss / (len(eval_data) - 1)
  
  ```
  
  **Tags:** transformer, pytorch, cross-entropy, evaluation-functions
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-11-16. Score: 3
  
  That is because the default reduction for the CEP loss calculation is mean. Hence to find the true average across all batches, you first multiply by the batch size and then divide by total number of data points. Note this only matters when all the batches are not equal, i.e. total number of data points do not perfectly divide batch size. Else you can simply add all the batch losses and divide by number of batches.
  
  
  ---
  
  