Answers.Body: <p>Fundamentally, you are seeing a difference in behavior during training v.s. during evaluation. The most typical reason for it is the difference in behavior of some <code>nn</code> layers that your library (pytorch) provides, depending on the mode that you are in.</p>
  <p>Check out documentation for <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=train#torch.nn.Module.train" rel="nofollow noreferrer">torch.nn.Module.train</a>.</p>
  <p>Most notably <code>nn.Dropout</code> and <code>nn.BatchNorm</code> layers are prone to that.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-08T23:41:23.360Z
Answers.Text: I was suspecting the dropout and batch norm layers to be the reason behind the discrepancy as well. When I update model.eval() to model.train(), I get lower errors. But I believe that the removal of batch norm and dropout layers should help in improving the loss during evaluation instead of degrading it. With dropout, some information bearing nodes are randomly switched off which is not the case during evaluation. Doesn't it mean a lower (or at worst the same) loss during evaluation as compared to training?
Answers.UserDisplayName: Nitish Agarwal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T11:53:11.557Z
Answers.Text: @NitishAgarwal As I understand, you've mentioned that you are strongly overfitting during training? Dropout can be understood as way to do model averaging. When averaging several overfitted models you'll must likely get a worse-performing one.
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-08T19:40:53.870Z
Answers.LastActivityDate: 2021-04-08T19:40:53.870Z
Answers.UserDisplayName: Kostya
Body: <pre><code>    def forward(self, image, proj, proj_inv):
          return self.predict_2d_joint_locations(image, proj, proj_inv)
  
      def criterion(self, predicted, gt):
          return self.mse(predicted, gt)
  
      def training_step(self, batch, batch_idx):
          player_images, j2d, j3d, proj, proj_inv, is_synth = batch
          predicted_2d_joint_locations = self.predict_2d_joint_locations(player_images, proj, proj_inv)
          train_loss = self.criterion(predicted_2d_joint_locations, j2d)
          self.log('train_loss', train_loss)
          return train_loss
  
      def validation_step(self, batch, batch_idx):
          player_images, j2d, j3d, proj, proj_inv, is_synth = batch
          predicted_2d_joint_locations = self.predict_2d_joint_locations(player_images, proj, proj_inv)
          val_loss = self.criterion(predicted_2d_joint_locations, j2d)
          self.log('val_loss', val_loss)
          return val_loss
  </code></pre>
  <p>I have this simple code for <code>training_step()</code> and <code>forward()</code> in Pytorch. Both the functions essentially do the same.</p>
  <p>Owing to a relatively small dataset, my model grossly overfits on the training data (as is evident from there being an orders of magnitude of difference between the training and validation losses). But that's fine for now, I am perfectly aware of that and will add more data soon.</p>
  <p>What surprises me is when I try to evaluate (infer). I don't have a separate test set (for now) and only have a training and a validation set. When I evaluate on the validation set, the mean squared error turns out to be in the same range as the validation loss my model is based on as expected. However, when I evaluate on the training set, the mean squared error I get is again in the same range as the validation loss (not the training loss).</p>
  <pre><code>    if args.val:
              check_dl = dataset.val_dataloader()
          else:
              check_dl = dataset.train_dataloader()
  
          for player_images,j2d,j3d,proj,proj_inv,is_synth in check_dl:
              if args.visualize:
                  # visualize dataset
                  player_images = player_images.cpu().numpy()
                  j2d_predicted = model(torch.from_numpy(player_images), proj, proj_inv).cpu().detach().numpy()
                  print(((j2d - j2d_predicted) ** 2).mean(), model.training_step((torch.from_numpy(player_images),j2d,j3d,proj,proj_inv,is_synth), 0))
  </code></pre>
  <p>When I print <code>print(((j2d - j2d_predicted) ** 2).mean()</code> for images in the training set after fetching the model from the trained checkpoint, I get numbers in the range of the validation loss. I retried the same by printing the loss using the <code>training_step()</code> function, but I again receive high losses (in the validation loss range).</p>
  <p><strong>Note:</strong> The inference mean squared errors I receive on the training set are high but they are not as high as when the training actually started. So, the pre-trained model is fetched properly. On a model with completely random weights, I should have received orders of magnitudes of higher errors. So, the model is definitely fetched correctly.</p>
  <p>I have been scratching my head over this. Any help would be really appreciated.</p>
  
Tags: <pytorch>
Title: Pytorch - Evaluation loss on the training set higher than loss during training
UserDisplayName: Nitish Agarwal
ThreadText: # Pytorch - Evaluation loss on the training set higher than loss during training
  
  (Asked by: username_0 on 2021-04-06)
  
  def forward(self, image, proj, proj_inv):
          return self.predict_2d_joint_locations(image, proj, proj_inv)
  
      def criterion(self, predicted, gt):
          return self.mse(predicted, gt)
  
      def training_step(self, batch, batch_idx):
          player_images, j2d, j3d, proj, proj_inv, is_synth = batch
          predicted_2d_joint_locations = self.predict_2d_joint_locations(player_images, proj, proj_inv)
          train_loss = self.criterion(predicted_2d_joint_locations, j2d)
          self.log('train_loss', train_loss)
          return train_loss
  
      def validation_step(self, batch, batch_idx):
          player_images, j2d, j3d, proj, proj_inv, is_synth = batch
          predicted_2d_joint_locations = self.predict_2d_joint_locations(player_images, proj, proj_inv)
          val_loss = self.criterion(predicted_2d_joint_locations, j2d)
          self.log('val_loss', val_loss)
          return val_loss
  
  I have this simple code for training_step() and forward() in Pytorch. Both the functions essentially do the same.
  Owing to a relatively small dataset, my model grossly overfits on the training data (as is evident from there being an orders of magnitude of difference between the training and validation losses). But that's fine for now, I am perfectly aware of that and will add more data soon.
  What surprises me is when I try to evaluate (infer). I don't have a separate test set (for now) and only have a training and a validation set. When I evaluate on the validation set, the mean squared error turns out to be in the same range as the validation loss my model is based on as expected. However, when I evaluate on the training set, the mean squared error I get is again in the same range as the validation loss (not the training loss).
      if args.val:
              check_dl = dataset.val_dataloader()
          else:
              check_dl = dataset.train_dataloader()
  
          for player_images,j2d,j3d,proj,proj_inv,is_synth in check_dl:
              if args.visualize:
                  # visualize dataset
                  player_images = player_images.cpu().numpy()
                  j2d_predicted = model(torch.from_numpy(player_images), proj, proj_inv).cpu().detach().numpy()
                  print(((j2d - j2d_predicted) ** 2).mean(), model.training_step((torch.from_numpy(player_images),j2d,j3d,proj,proj_inv,is_synth), 0))
  
  When I print print(((j2d - j2d_predicted) ** 2).mean() for images in the training set after fetching the model from the trained checkpoint, I get numbers in the range of the validation loss. I retried the same by printing the loss using the training_step() function, but I again receive high losses (in the validation loss range).
  Note: The inference mean squared errors I receive on the training set are high but they are not as high as when the training actually started. So, the pre-trained model is fetched properly. On a model with completely random weights, I should have received orders of magnitudes of higher errors. So, the model is definitely fetched correctly.
  I have been scratching my head over this. Any help would be really appreciated.
  
  **Tags:** pytorch
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-08. Score: 0
  
  Fundamentally, you are seeing a difference in behavior during training v.s. during evaluation. The most typical reason for it is the difference in behavior of some `nn` layers that your library (pytorch) provides, depending on the mode that you are in.
  
  Check out documentation for [torch.nn.Module.train](https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=train#torch.nn.Module.train).
  
  Most notably `nn.Dropout` and `nn.BatchNorm` layers are prone to that.
  
  > **Comments:**
  > **username_0**: I was suspecting the dropout and batch norm layers to be the reason behind the discrepancy as well. When I update model.eval() to model.train(), I get lower errors. But I believe that the removal of batch norm and dropout layers should help in improving the loss during evaluation instead of degrading it. With dropout, some information bearing nodes are randomly switched off which is not the case during evaluation. Doesn't it mean a lower (or at worst the same) loss during evaluation as compared to training?  
  > **username_1**: @NitishAgarwal As I understand, you've mentioned that you are strongly overfitting during training? Dropout can be understood as way to do model averaging. When averaging several overfitted models you'll must likely get a worse-performing one.  
  
  ---
  
  