So, what I understand is that if you pass input x to model in case of model.train() everything is ok but for the same input x, model.eval() is giving worse result. model.eval() only affect layers such as batchnorm and dropout, so I guess you have such layers in your model definition. I think this post in that case can help alot as this is a tricky challenge sometimes:

  
    
    
    Model.eval() gives incorrect loss for model with batchnorm layers 
  
  
    The testing code which is called every 100 training iters is as following: 
def test_model(model):
    psnr_test_avg = 0
    loss_test_avg = 0
    model.eval()
    
    for iteration, test_data in enumerate(dataloader_test, 0):
        label_test = test_data['label_patch']
        residue_test = test_data['residue_patch']
        stacked_test = test_data['stacked_patch']
        microshift_test = test_data['train_patch']
        inputs_test, residues_test, labels_test, microshifts_test = Variabl…
  


There are few tricks to solve issue in the aforementioned thread. I hope it helps.
Bests