Looking at the code 18 it does look like distributed scatters the input! this is very inefficient when inputs are images for example. It would be much more efficient for each process to read its portion from a shared directory/file.
I think I would subclass DistributedDataParallel and do just that.
How is the performance of distributed pytorch with large inputs on multi-gpu (say 32+ nodes)?