mcarilli:

One simple thing you can try is, don’t use the dataloader at all. In each DDP process, create a single dummy batch of data and use that through all the timing iterations.


Thanks for the suggestion I will try that. Is there some rough number on how fast my dataloder should be in seconds or relative to the model?
Also thanks to everyone for helping out and explaining!