I am trying to train GNN using DDP multi-node GPUs. I am using pytorch 1.7 and gloo as backend. I get an error like below on machine 2.
-- Process 0 terminated with the following error:
Traceback (most recent call last):
......
  File "/home/karthi/venvv/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 532, in _ddp_init_helper
    self.gradient_as_bucket_view)
RuntimeError: replicas[0][0] in this process with sizes [64, 5] appears not to match sizes of the same param in process 0.

But the same code works in single machine multi-gpu DDP. Please help me.This text will be hidden