NCCL_DEBUG=INFO won’t give any useful information. NCCL_DEBUG_SUBSYS=ALL will give some memory alloc information which doesn’t help too much.
  File "train.py", line 226, in train
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:401 Mem Alloc Size 4 pointer 0x7fa25c0a1210
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:402 Mem Alloc Size 4 pointer 0x7fa25c0a11f0
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:452 Mem Alloc Size 4 pointer 0x7fa25c0a1170
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:401 Mem Alloc Size 4 pointer 0x7fa25c0a11f0
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:402 Mem Alloc Size 4 pointer 0x7fa25c0a1210
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:452 Mem Alloc Size 4 pointer 0x7fa25c0a1170
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:401 Mem Alloc Size 4 pointer 0x7fa25c0a1210
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:402 Mem Alloc Size 4 pointer 0x7fa25c0a11f0
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:452 Mem Alloc Size 4 pointer 0x7fa25c0a1170
    ddp_model = DistributedDataParallel(model, device_ids=[local_rank], output_device=local_rank)
  File "/mnt/anaconda/envs/pytorch/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 578, in __init__
slurm-2:2498277:2498586 [0] NCCL INFO graph/search.cc:402 Mem Alloc Size 4 pointer 0x7f47040a11d0
slurm-2:2498277:2498586 [0] NCCL INFO graph/search.cc:452 Mem Alloc Size 4 pointer 0x7f47040a1170
slurm-2:2498277:2498586 [0] NCCL INFO graph/search.cc:401 Mem Alloc Size 4 pointer 0x7f47040a11d0
slurm-2:2498277:2498586 [0] NCCL INFO graph/search.cc:402 Mem Alloc Size 4 pointer 0x7f47040a1210
    dist._verify_model_across_ranks(self.process_group, parameters)
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:401 Mem Alloc Size 4 pointer 0x7fa25c0a11f0
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:402 Mem Alloc Size 4 pointer 0x7fa25c0a1210
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:452 Mem Alloc Size 4 pointer 0x7fa25c0a1170
RuntimeError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:957, unhandled system error, NCCL version 21.0.3
ncclSystemError: System call (socket, malloc, munmap, etc) failed.