Answers.Body: <p>You want to compute the <strong>mean</strong> loss over all batches. What you need to do is to divide the sum of batch losses with the number of batches!</p>
  
  <h3>In your case:</h3>
  
  <p>You have a training set of <span class="math-container">$21700$</span> samples and a batch size of <span class="math-container">$500$</span>. This means that you take <span class="math-container">$21700/500 \approx 43$</span> training iterations. This means that for each epoch the model is updated <span class="math-container">$43$</span> times! So the way you compute your training loss, that is what you need to divide by. </p>
  
  <p>Note: I'm not sure what exactly you're trying to plot but I'm assuming you want to plot the training losses and the validation losses</p>
  
  
  
  <pre><code>training_loss = []
  validation_loss = []
  training_steps = num_samples // batch_size
  validation_steps = num_validation_samples // batch_size
  
  for epoch in range(num_epochs):
  
      # Training steps
      total_loss = 0
      for b in range(training_steps):
          batch_loss = ...  # compute batch loss
          total_loss += batch_loss
      training_loss.append(total_loss / training_steps)
  
      # Validation steps
      total_loss = 0
      for b in range(validation_steps):
          batch_loss = ...  # compute batch validation loss
          total_loss += batch_loss
      training_loss.append(total_loss / validation_steps)
  
  # Plot training and validation curves
  plt.plot(range(num_epochs), training_loss)
  plt.plot(range(num_epochs), validation_loss)
  </code></pre>
  
  <p>Another way would be to store the losses in a list and compute the mean. You can use this if you're not sure with what to divide.</p>
  
  
  
  <pre><code>...
  
  for epoch in range(num_epochs):
  
      list_of_batch_losses = []  # initialize list that is going to store batch losses
  
      # Training steps
      for b in range(training_steps):
          batch_loss = ...  # compute batch loss
          list_of_batch_losses.append(batch_loss)  # store loss in a list
  
      epoch_loss = np.mean(list_of_batch_losses)
      training_loss.append(epoch_loss)
  
      ...
  
  plt.plot(range(num_epochs), training_loss)
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-31T20:56:39.457Z
Answers.Text: Yup! I wanted to print the training and validation losses. Earlier, i was dividing by number of samples for training and validation respectively, and i they had very different scales, so my training loss looked much much greater than my validation loss. But yeah, what you said is right and worked for me. Thank you :-) .
Answers.UserDisplayName: Burple
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-31T16:36:22.923Z
Answers.LastActivityDate: 2019-07-31T16:36:22.923Z
Answers.UserDisplayName: Djib2011
Body: <p>I am using a CNN to train on some data, where training size = 21700 samples, and test size is 653 samples, and say I am using a batch_size of 500 (I am accounting for samples out of batch size as well). </p>
  
  <p>I have been looking this up for a long time now, but can't get a clear answer, but when plotting the loss functions to check for whether the model is overfitting or not, do I plot as follows </p>
  
  <pre><code>for j in range(num_epochs):
    &lt;some training code---Take gradient descent step do wonders&gt;
    batch_loss=0
    for i in range(num_batches_train):
         batch_loss = something....criterion(target,output)...
         total_loss += batch_loss
    Losses_Train_Per_Epoch.append(total_loss/num_samples_train)#and this is 
  </code></pre>
  
  <p>where I need help</p>
  
  <pre><code>Losses_Train_Per_Epoch.append(total_loss/num_batches_train)
  and doing the same for Losses_Validation_Per_Epoch.
  plt.plot(Losses_Train_Per_Epoch, Losses_Validation_Per_epoch)
  </code></pre>
  
  <p>So, basically, what I am asking is, should I divide by num_samples or num_batches or batch_size? Which one is it?</p>
  
Tags: <convolutional-neural-networks><objective-functions><supervised-learning><pytorch>
Title: Are the training loss and validation loss plotted per sample or per batch?
UserDisplayName: Burple
ThreadText: # Are the training loss and validation loss plotted per sample or per batch?
  
  (Asked by: username_0 on 2019-07-31)
  
  I am using a CNN to train on some data, where training size = 21700 samples, and test size is 653 samples, and say I am using a batch\_size of 500 (I am accounting for samples out of batch size as well).
  
  I have been looking this up for a long time now, but can't get a clear answer, but when plotting the loss functions to check for whether the model is overfitting or not, do I plot as follows
  
  ```
  for j in range(num_epochs):
    <some training code---Take gradient descent step do wonders>
    batch_loss=0
    for i in range(num_batches_train):
         batch_loss = something....criterion(target,output)...
         total_loss += batch_loss
    Losses_Train_Per_Epoch.append(total_loss/num_samples_train)#and this is 
  
  ```
  
  where I need help
  
  ```
  Losses_Train_Per_Epoch.append(total_loss/num_batches_train)
  and doing the same for Losses_Validation_Per_Epoch.
  plt.plot(Losses_Train_Per_Epoch, Losses_Validation_Per_epoch)
  
  ```
  
  So, basically, what I am asking is, should I divide by num\_samples or num\_batches or batch\_size? Which one is it?
  
  **Tags:** convolutional-neural-networks, objective-functions, supervised-learning, pytorch
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-07-31. Score: 3 | Accepted answer
  
  You want to compute the **mean** loss over all batches. What you need to do is to divide the sum of batch losses with the number of batches!
  
  ### In your case:
  
  You have a training set of $21700$ samples and a batch size of $500$. This means that you take $21700/500 \approx 43$ training iterations. This means that for each epoch the model is updated $43$ times! So the way you compute your training loss, that is what you need to divide by.
  
  Note: I'm not sure what exactly you're trying to plot but I'm assuming you want to plot the training losses and the validation losses
  
  ```
  training_loss = []
  validation_loss = []
  training_steps = num_samples // batch_size
  validation_steps = num_validation_samples // batch_size
  
  for epoch in range(num_epochs):
  
      # Training steps
      total_loss = 0
      for b in range(training_steps):
          batch_loss = ...  # compute batch loss
          total_loss += batch_loss
      training_loss.append(total_loss / training_steps)
  
      # Validation steps
      total_loss = 0
      for b in range(validation_steps):
          batch_loss = ...  # compute batch validation loss
          total_loss += batch_loss
      training_loss.append(total_loss / validation_steps)
  
  # Plot training and validation curves
  plt.plot(range(num_epochs), training_loss)
  plt.plot(range(num_epochs), validation_loss)
  
  ```
  
  Another way would be to store the losses in a list and compute the mean. You can use this if you're not sure with what to divide.
  
  ```
  ...
  
  for epoch in range(num_epochs):
  
      list_of_batch_losses = []  # initialize list that is going to store batch losses
  
      # Training steps
      for b in range(training_steps):
          batch_loss = ...  # compute batch loss
          list_of_batch_losses.append(batch_loss)  # store loss in a list
  
      epoch_loss = np.mean(list_of_batch_losses)
      training_loss.append(epoch_loss)
  
      ...
  
  plt.plot(range(num_epochs), training_loss)
  
  ```
  
  > **Comments:**
  > **username_0**: Yup! I wanted to print the training and validation losses. Earlier, i was dividing by number of samples for training and validation respectively, and i they had very different scales, so my training loss looked much much greater than my validation loss. But yeah, what you said is right and worked for me. Thank you :-) .  
  
  ---
  
  