The difference in loss with the model in eval() vs train() mode might be happening if you have any batchnorm in your dataset. In the inference stage, the behaviour is per-image for batch norm as far as I know.