Wait what? Did something change from your first post? Are you now saying that the first 1-25 batches take 0.4s and the batches 25-50 take 2-3 seconds? So it’s slower as you go now? What happens at 50-75 batches, or 75-100?
How many batches does your dataset consist of? 100 validation images with a batch size of 1 = 100 batches?