Hey! I am trying to implement DDP + MDP and followed [this].(Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.10.0+cu102 documentation 2)
I’m having issues with the demo code itself so any help is welcome! I’m trying to run it on a p2.8xlarge gpu instance on AWS and using the pytorch 1.9.0 version.
Here’s the error that I’m getting:
ip-172-31-136-108:121:121 [0] NCCL INFO Bootstrap : Using [0]ecs-eth0:169.254.172.12<0> [1]eth0:172.31.136.108<0>
ip-172-31-136-108:121:121 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ip-172-31-136-108:121:121 [0] NCCL INFO NET/IB : No device found.
ip-172-31-136-108:121:121 [0] NCCL INFO NET/Socket : Using [0]ecs-eth0:169.254.172.12<0> [1]eth0:172.31.136.108<0>
ip-172-31-136-108:121:121 [0] NCCL INFO Using network Socket
NCCL version 2.7.8+cuda10.2
net2
ip-172-31-136-108:122:122 [0] NCCL INFO Bootstrap : Using [0]ecs-eth0:169.254.172.12<0> [1]eth0:172.31.136.108<0>
ip-172-31-136-108:122:122 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ip-172-31-136-108:122:122 [0] NCCL INFO NET/IB : No device found.
ip-172-31-136-108:122:122 [0] NCCL INFO NET/Socket : Using [0]ecs-eth0:169.254.172.12<0> [1]eth0:172.31.136.108<0>
ip-172-31-136-108:122:122 [0] NCCL INFO Using network Socket

ip-172-31-136-108:122:202 [0] init.cc:573 NCCL WARN Duplicate GPU detected : rank 1 and rank 0 both on CUDA device 170

ip-172-31-136-108:121:201 [0] init.cc:573 NCCL WARN Duplicate GPU detected : rank 0 and rank 1 both on CUDA device 170
ip-172-31-136-108:122:202 [0] NCCL INFO init.cc:840 -> 5
ip-172-31-136-108:121:201 [0] NCCL INFO init.cc:840 -> 5
ip-172-31-136-108:121:201 [0] NCCL INFO group.cc:73 -> 5 [Async thread]
ip-172-31-136-108:122:202 [0] NCCL INFO group.cc:73 -> 5 [Async thread]
Traceback (most recent call last):
  File "relnet/tools/test_mdp.py", line 78, in <module>
    run_demo(demo_model_parallel, world_size)
  File "relnet/tools/test_mdp.py", line 72, in run_demo
    mp.spawn(demo_fn, args=(world_size,), nprocs=world_size, join=True)
  File "/opt/pensa-recognition/pensa-cognition/relnet/.venv/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/opt/pensa-recognition/pensa-cognition/relnet/.venv/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
    while not context.join():
  File "/opt/pensa-recognition/pensa-cognition/relnet/.venv/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 150, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/opt/pensa-recognition/pensa-cognition/relnet/.venv/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/opt/pensa-recognition/pensa-cognition/relnet/relnet/tools/test_mdp.py", line 51, in demo_model_parallel
    ddp_mp_model = DDP(mp_model)
  File "/opt/pensa-recognition/pensa-cognition/relnet/.venv/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 496, in __init__
    dist._verify_model_across_ranks(self.process_group, parameters)
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:911, invalid usage, NCCL version 2.7.8

I’m only making GPU 0 and 1 visible and running the demo code as is. What am I doing wrong?
Thanks in advance!