maralm:

I am trying to load the cpu model in one thread and broadcast in gpu.


The default process group is per-process object, instead of per-thread. Is this just a typo and you actually mean “process”?
if args.local_rank not in [-1,0]:
    for name, p in self.encoder.layer[0].named_parameters():
        torch.distributed.broadcast(p, src=0)

For collective communications, it requires all ranks to make the same number of c10d API invocations in the same order. It seems that, with the above code, rank 0 is not participating in the broadcast? If you need broadcast in a subgroup, you will need to first create a subgroup using the new_group 1 API and then call boradcast in that group.