Surely I did this. Sorry for omitting this in the code above. It’s like:
...
torch.cuda.set_device(rank)
...
model = Net().cuda()
model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[rank], output_device=rank)
...

However, the parameters I got still differ.