Thank you. I am trying to load the cpu model in one thread and broadcast in gpu. However, I have an issue with the barrier. Here is my code:
if args.local_rank in [-1,0]:
    for net1,net2 in zip(self.encoder.layer[0].named_parameters(), model_cpu.bert.encoder.layer[i].named_parameters()):
         net1[1].data.copy_(net2[1].data.clone(), non_blocking=True)
 
torch.distributed.barrier()

if args.local_rank not in [-1,0]:
    for name, p in self.encoder.layer[0].named_parameters():
        torch.distributed.broadcast(p, src=0)

my code stucks at the barrier. Any idea what could be wrong with my code? Should I even use barrier?