Dear @mrshenli,
Following up on the Hogwild! paper mentioned previously. I also found out that Pytorch has the PytorchHogwild example using multiprocessing techniques.
I would want to redo this and then extend it for multi-machine and distributed training, either using existing Pytorch DDP or custom design built upon distributed communication APIs.
The big picture is shown below:
image737×555 20.2 KB
Within the Machine#0 !Hogwild can be performed. Does this make sense to you?
However, when I am making a simple example to test the torch.multiprocessing first locally, it seems that multiprocessing cannot work. Please see the result below:
import os
import torch
from torch import nn
import torch.distributed as dist
import torch.multiprocessing as mp
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(torch.__version__)

>> 1.4.0

class Net(nn.Module):
    def __init__(self):
        super().__init__() 
        self.out = nn.Linear(in_features=5, out_features=1)

    def forward(self, t):
        # output
        t = self.out(t)
        return t

def addNet(model):
    for para in model.parameters():
        tmp = torch.ones_like(para.data)
        para.data = para.data + tmp
        print(para.data)

torch.manual_seed(101)
model = Net()
model.share_memory()
for para in model.parameters():
    print(para.data)

>>> tensor([[-0.2701, -0.0445, -0.3659,  0.3463, -0.1884]])
>>> tensor([-0.4306])

num_processes = 2
processes = []
for rank in range(num_processes):
    p = mp.Process(target=addNet, args=(model,))
    p.start()
    processes.append(p)
for p in processes:
    p.join()

>>> tensor([[0.7299, 0.9555, 0.6341, 1.3463, 0.8116]])
>>> tensor([[0.7299, 0.9555, 0.6341, 1.3463, 0.8116]])
>>> tensor([0.5694])
>>> tensor([0.5694])

for para in model.parameters():
    print(para.data)

>>> tensor([[-0.2701, -0.0445, -0.3659,  0.3463, -0.1884]])
>>> tensor([-0.4306])

It seems to me that the children processes just made a copy of the NN in the parent process and the NN was not being shared among the parent and two children processes. Did I miss anything from PytorchHogwild? Or it will be very much appreciated if you could share any thoughts.
Best,
Ziyi