Hi,
I am training a model with DDP w/ 8 GPUs (one process per GPU) and the DataLoader with 8 data loading workers. Training is slow during the first epoch and speeds up significantly immediately starting the second epoch and onwards. Has anyone seen this before?