If calling model.eval() yields worse results on the same input data (or similar data), it could mean that e.g. the running stats of the batchnorm layers are off and you could change the momentum or check if you are dealing with data samples from different distributions.