I found a related issue. My data is stored in multiple np.memmap files, and I use a dataloader to access entries in these different np.memmap files. Dataloader shuffles indices at the start of every epoch.
At the beginning of every epoch, the data loader is slow, but then towards the end it speeds up quite a bit. I wonder whether this is because the dataloader clears all the data loaded to main memory at the end of every epoch.