Solved by midhunharikumar in post #2 
              
              
                Nodes in this context refers to the physical machine or VM. nn.DataParallel performs model replication and data distribution within the GPU’s connected to a single machine. All operations happen within a single system process. There is no multi-process synchronization. nn.DistributedDataParallel can…