Solved by ptrblck in post #2 
              
              
                Did you try to use different seeds without the validation loop and check how the training behaves? 
I guess your training might not be very stable so that calling into the pseudo-random number generator during validation would change the next random values and your model would diverge during the tra…