Hey @Musoy_King, looks like NCCL broadcast crashed. Can you try if directly calling dist.broadcast 1 would fail too?

  

      github.com
  

  
    pytorch/pytorch/blob/c65f332da47eb9bc76aefc50122cae2630fff2cc/torch/nn/parallel/distributed.py#L622 2



      
      
              module_states = []
              for name, param in self.module.named_parameters():
                  if name not in self.parameters_to_ignore:
                      module_states.append(param.detach())
          
          
    for name, buffer in self.module.named_buffers():
                  if name not in self.parameters_to_ignore:
                      module_states.append(buffer.detach())
          
          
    if len(module_states) > 0:
                  self._distributed_broadcast_coalesced(
                      module_states, self.broadcast_bucket_size, authoritative_rank
                  )
          
          
def _log_and_throw(self, err_type, err_msg):
              if self.logger is not None:
                  self.logger.set_error_and_log(f"{str(err_type)}: {err_msg}")
              raise err_type(err_msg)
          
          
def _ddp_init_helper(
              self, parameters, expect_sparse_gradient, param_to_name_mapping
      
    
  


  

  
    
    
  

  


Also, looks like you are using ipython or notebook. Can you try to directly use python to run the script on the two nodes?