I’m working on a project using 20bn something something dataset. Somehow, I run into exactly the same issue with a baseline 4, which has a pytorch dataloader. The second epoch and onwards, the performance slows down ~2x. I’m going to try the dataloader you suggested, but I’m still curious why pytorch dataloader has this issue. Does anyone has a clue? Thanks in advance! @albanD