How large is your batch size when you are using the DataLoader?
Each worker of the DataLoader will create a full batch before the DataLoader loop is executed, so in particular the first iteration might be slow, since all workers would start collecting all samples. Once the workers are done, the ready batches will be added to a queue and the workers will start creating the next batch while the training loop is executed, which might reduce the data loading time assuming that the actual model training is not tiny compared to the data loading.