Distributed collectives such as broadcast/allreduce can help you initialize them in the same way across different workers.
For example, you can use broadcast to make the value the same as rank 0’s across all ranks, or allreduce to take the mean value across all ranks. The different collectives are documented here: Distributed communication package - torch.distributed — PyTorch 1.10.0 documentation 1