This might be happening if you are using e.g. batchnorm layers, with “bad” running stats for the mean and var. If you use the model in training mode during evaluation, the batch statistics will be used and the running stats updated (which could be seen as a data leak for future model.eval() validation runs).
You could play around with the momentum to smooth the update of the running stats and check other posts in this forum, as other users are also facing this issue.