Will using nn.DistributedDataParallel will increase the GPU utilisation or IO utilisation.
I varied the number of workers from 0 to 32, and surprisingly the Disk IO utilisation is the same throughout. Also, time takes to run 1 epoch is fastest, when the number of workers is 0.