Validation loop should not be effecting the model at all ideally.
Because you only do feed forward in that case. Make sure you are using model.eval() or torch.no_grad() contextmanager when evaluating.