My first two guesses would be:


The dataset needs to refill the shuffle buffer after each epoch, like model.fit(ds.shuffle(buffer_size).repeat()) instead of model.fit(ds.repeat().shuffle(), steps_per_epoch=N)


Maybe something with the evaluation logic?