Thank you for the quick reply. Yes the seeds are all set. The loss curves are overlapping so the batches are deterministic.
I also repeated experiments multiple times with different model sizes and data, all have similar phenomenon. The gap is just too big to be the result of randomness.