If you don’t switch to .eval() before using the evaluation dataset (or test set), the batch norm stats will be updated, so your model will “train” using this dataset, which is considered a data leakage.
Shuffling the validation or test dataset might even result in “better” stats to leak.
Note that shuffling does not change anything, if you set the model properly to eval() before validating the model.