This problem may be due to the “batch normalization”. When you are evaluating your model, you should disable batch normalization. Use “model.eval()” when you want to evaluate the model (so batch normalization will be disabled) and use “model.train()” again when you want train the model.