Answers.Body: <p>You want to compute the <strong>mean</strong> loss over all batches. What you need to do is to divide the sum of batch losses with the number of batches!</p>
  
  <h3>In your case:</h3>
  
  <p>You have a training set of <span class="math-container">$21700$</span> samples and a batch size of <span class="math-container">$500$</span>. This means that you take <span class="math-container">$21700/500 \approx 43$</span> training iterations. This means that for each epoch the model is updated <span class="math-container">$43$</span> times! So the way you compute your training loss, that is what you need to divide by. </p>
  
  <p>Note: I'm not sure what exactly you're trying to plot but I'm assuming you want to plot the training losses and the validation losses</p>
  
  
  
  <pre><code>training_loss = []
  validation_loss = []
  training_steps = num_samples // batch_size
  validation_steps = num_validation_samples // batch_size
  
  for epoch in range(num_epochs):
  
      # Training steps
      total_loss = 0
      for b in range(training_steps):
          batch_loss = ...  # compute batch loss
          total_loss += batch_loss
      training_loss.append(total_loss / training_steps)
  
      # Validation steps
      total_loss = 0
      for b in range(validation_steps):
          batch_loss = ...  # compute batch validation loss
          total_loss += batch_loss
      training_loss.append(total_loss / validation_steps)
  
  # Plot training and validation curves
  plt.plot(range(num_epochs), training_loss)
  plt.plot(range(num_epochs), validation_loss)
  </code></pre>
  
  <p>Another way would be to store the losses in a list and compute the mean. You can use this if you're not sure with what to divide.</p>
  
  
  
  <pre><code>...
  
  for epoch in range(num_epochs):
  
      list_of_batch_losses = []  # initialize list that is going to store batch losses
  
      # Training steps
      for b in range(training_steps):
          batch_loss = ...  # compute batch loss
          list_of_batch_losses.append(batch_loss)  # store loss in a list
  
      epoch_loss = np.mean(list_of_batch_losses)
      training_loss.append(epoch_loss)
  
      ...
  
  plt.plot(range(num_epochs), training_loss)
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-31T20:56:39.457Z
Answers.Text: Yup! I wanted to print the training and validation losses. Earlier, i was dividing by number of samples for training and validation respectively, and i they had very different scales, so my training loss looked much much greater than my validation loss. But yeah, what you said is right and worked for me. Thank you :-) .
Answers.UserDisplayName: Burple
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-31T16:36:22.923Z
Answers.LastActivityDate: 2019-07-31T16:36:22.923Z
Answers.UserDisplayName: Djib2011
Body: <p>I am using a CNN to train on some data, where training size = 21700 samples, and test size is 653 samples, and say I am using a batch_size of 500 (I am accounting for samples out of batch size as well). </p>
  
  <p>I have been looking this up for a long time now, but can't get a clear answer, but when plotting the loss functions to check for whether the model is overfitting or not, do I plot as follows </p>
  
  <pre><code>for j in range(num_epochs):
    &lt;some training code---Take gradient descent step do wonders&gt;
    batch_loss=0
    for i in range(num_batches_train):
         batch_loss = something....criterion(target,output)...
         total_loss += batch_loss
    Losses_Train_Per_Epoch.append(total_loss/num_samples_train)#and this is 
  </code></pre>
  
  <p>where I need help</p>
  
  <pre><code>Losses_Train_Per_Epoch.append(total_loss/num_batches_train)
  and doing the same for Losses_Validation_Per_Epoch.
  plt.plot(Losses_Train_Per_Epoch, Losses_Validation_Per_epoch)
  </code></pre>
  
  <p>So, basically, what I am asking is, should I divide by num_samples or num_batches or batch_size? Which one is it?</p>
  
Tags: <convolutional-neural-networks><objective-functions><supervised-learning><pytorch>
Title: Are the training loss and validation loss plotted per sample or per batch?
UserDisplayName: Burple
ThreadText: # Are the training loss and validation loss plotted per sample or per batch?
  
  (Asked by: username_0 on 2019-07-31)
  
  I am using a CNN to train on some data, where training size = 21700 samples, and test size is 653 samples, and say I am using a batch_size of 500 (I am accounting for samples out of batch size as well). 
  I have been looking this up for a long time now, but can't get a clear answer, but when plotting the loss functions to check for whether the model is overfitting or not, do I plot as follows 
  for j in range(num_epochs):
    
    batch_loss=0
    for i in range(num_batches_train):
         batch_loss = something....criterion(target,output)...
         total_loss += batch_loss
    Losses_Train_Per_Epoch.append(total_loss/num_samples_train)#and this is 
  
  where I need help
  Losses_Train_Per_Epoch.append(total_loss/num_batches_train)
  and doing the same for Losses_Validation_Per_Epoch.
  plt.plot(Losses_Train_Per_Epoch, Losses_Validation_Per_epoch)
  
  So, basically, what I am asking is, should I divide by num_samples or num_batches or batch_size? Which one is it?
  
  **Tags:** convolutional-neural-networks, objective-functions, supervised-learning, pytorch
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-07-31. Score: 3 | Accepted answer
  
  You want to compute the mean loss over all batches. What you need to do is to divide the sum of batch losses with the number of batches!
  In your case:
  You have a training set of $21700$ samples and a batch size of $500$. This means that you take $21700/500 \approx 43$ training iterations. This means that for each epoch the model is updated $43$ times! So the way you compute your training loss, that is what you need to divide by. 
  Note: I'm not sure what exactly you're trying to plot but I'm assuming you want to plot the training losses and the validation losses
  training_loss = []
  validation_loss = []
  training_steps = num_samples // batch_size
  validation_steps = num_validation_samples // batch_size
  
  for epoch in range(num_epochs):
  
      # Training steps
      total_loss = 0
      for b in range(training_steps):
          batch_loss = ...  # compute batch loss
          total_loss += batch_loss
      training_loss.append(total_loss / training_steps)
  
      # Validation steps
      total_loss = 0
      for b in range(validation_steps):
          batch_loss = ...  # compute batch validation loss
          total_loss += batch_loss
      training_loss.append(total_loss / validation_steps)
  
  # Plot training and validation curves
  plt.plot(range(num_epochs), training_loss)
  plt.plot(range(num_epochs), validation_loss)
  
  Another way would be to store the losses in a list and compute the mean. You can use this if you're not sure with what to divide.
  ...
  
  for epoch in range(num_epochs):
  
      list_of_batch_losses = []  # initialize list that is going to store batch losses
  
      # Training steps
      for b in range(training_steps):
          batch_loss = ...  # compute batch loss
          list_of_batch_losses.append(batch_loss)  # store loss in a list
  
      epoch_loss = np.mean(list_of_batch_losses)
      training_loss.append(epoch_loss)
  
      ...
  
  plt.plot(range(num_epochs), training_loss)
  
  > **Comments:**
  > **username_0**: Yup! I wanted to print the training and validation losses. Earlier, i was dividing by number of samples for training and validation respectively, and i they had very different scales, so my training loss looked much much greater than my validation loss. But yeah, what you said is right and worked for me. Thank you :-) .  
  
  ---
  
  