Why my DistributedDataParallel is slower than DataParallel if my Dataset is not loaded fully in memory distributed
  
  
    Is it possible, that if your data is small enough to entirely fit into the memory, the DDP setup overhead is just increasing time on the task without any performance improvement? In other words: GPU utilization is small enough, you just can’t see the gains of using multiple GPUs
  


Good question @Alexey_Demyanchuk.
But my answer is no. My data is large to fit in GPU, that is why I started by loading it from disk. However, when I found that DDP is slower than DP, as I mentioned in the question, I started comparing both with different configurations to see what works. Eventually, the only change that made DDP faster is when I reduced my data size and loaded it into memory. I hope this clarifies the situation.