first I suggest to evaluate the model on testset.
you can try and see if there is a difference if when you evaluate you use with torch.no_grad() instead on switching to eval mode
however no reason to perform inference in training mode