You can refer to the following example.
You can change the model weights to a Tensor by parameters_to_vector function, communicate between ranks, revert that to weights by vector_to_parameters funciton.
I’m using a collective communication function to synchronize weight parameters on all GPUs but you can change that to point-to-point communication functions such as torch.distributed.send and torch.distributed.recv.
docs: torch.distributed 1
import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel

from torch.nn.utils import parameters_to_vector, vector_to_parameters

model = ...

# synchronize model parameters across nodes
vector = parameters_to_vector(model.parameters())

dist.broadcast(vector, 0)   # broadcast parameters to other processes
if dist.get_rank() != 0:
    vector_to_parameters(vector, model.parameters())