I was trying to start a distributed training across 2 machines using RoCE.
Environment:

Ubuntu 20.04
pytorch torch==1.10.1+cu113

The task is established by slurm:
#!/bin/bash
#SBATCH --gres=gpu:2
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=4

srun python train.py

And the error I met is:
RuntimeError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:957, unhandled system error, NCCL version 21.0.3
ncclSystemError: System call (socket, malloc, munmap, etc) failed.```

Line 957 in pytorch/ProcessGroupNCCL.cpp at v1.10.1 · pytorch/pytorch · GitHub 2 is:
C10D_NCCL_CHECK(ncclGroupEnd(), c10::nullopt);

Training works if I set ‘NCCL_IB_DISABLE’ = ‘1’. Which might indicate this might be a RoCE related issue.
I was wonder if I’m looking the correct place. And could anyone help me with this issue?