Using model_fit on the validation dataset seems wrong and I assume that you might be optimizing the model internally.
If so, make sure to calculate the forward passes only to get the predictions and to compute the accuracy of the model. No learning rate scheduler or optimizer would be needed to do so.