I was thinking to load the model on cpu in one process, transfer it to the gpu 0 (which is in the same process) and from gpu 0, copy weights to other gpus in other processes. Broadcasting is the correct way to do so?

Yes, this looks correct to me. DistributedDataParallel 6 is actually doing the similar thing in its constructor.