@C_Ashraf Is the data loading time for your case really slow or the time to actually execute DistributedDataParallel? If you have a small self contained example demonstrating the problem, it would be easier to narrow down the issue.