How many nodes are you using and how many GPUs per node? Also, which communication backend are you using?
Also, it might be helpful to debug if you could share the code you’re using to initialize and train using DistributedDataParallel?