Did you try to use different seeds without the validation loop and check how the training behaves?
I guess your training might not be very stable so that calling into the pseudo-random number generator during validation would change the next random values and your model would diverge during the training.
Here 4 is an explanation of the RNG behavior during the validation loop, if a DataLoader is used with a workatround. Also here 3 is a similar issue to yours.