Hi, guys. I’m trying to build a simple distributed data parallel training program with 1 GPU per process. Firstly I followed https://pytorch.org/tutorials/intermediate/dist_tuto.html 2 and added some modification.
def run(rank, size):
    ...
    model = Net()
    ...

if __name__ == '__main__':
    ...
    for rank in range(size):
        p = Process(target=init_process, args=(rank, size, run))
        ...

However, after reading the example https://github.com/pytorch/examples/tree/master/mnist_hogwild 4, I found model is one of the arguments of a process:
if __name__ == '__main__':
    ...
    model = Net().to(device)
    ...
    for rank in range(args.num_process):
        p = mp.Process(target=train, args=(rank, args, model, device, dataloader_kwargs))
        ...

So I just wonder: where should the model be declared, inside or outside these processes? When and how do the gradients get synchronized?