Hi there! I’m currently using DDP to initialize a job on 32 compute nodes but it seems to be failing as not all workers are joining in even though the script is successfully running on all nodes. Each task starts successfully but then it seems only certain ranks are actually joining in during dist.init_process_group, given the waiting message 2022-01-06,00:00:41 | INFO | Rank 15 | Waiting in store based barrier to initialize process group for rank: 15, key: store_based_barrier_key:1 (world_size=128, worker_count=43, timeout=0:30:00).
My current slurm script looks like:
#SBATCH --account=cstdl
#SBATCH --nodes=32
#SBATCH --gres=gpu:4
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=24
#SBATCH --time=00:05:00
#SBATCH --partition=booster
#SBATCH --job-name=open_clip_debugging

# load low-level libraries
ml purge
eval "$(/p/project/ccstdl/gordon2/miniconda3/bin/conda shell.bash hook)" # init conda
conda activate open_clip
ml use $OTHERSTAGES
ml GCC/10.3.0
ml NCCL/2.10.3-1-CUDA-11.3
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL
export NCCL_IB_TIMEOUT = 22
export NCCL_IGNORE_CPU_AFFINITY=1
export CUDA_VISIBLE_DEVICES=0,1,2,3
export MASTER_PORT=12802
export WORLD_SIZE=128

### get the first node name as master address - customized for vgg slurm
### e.g. master(gnodee[2-5],gnoded1) == gnodee2
echo "NODELIST="${SLURM_NODELIST}
master_addr=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n 1)
export MASTER_ADDR=$master_addr
echo "MASTER_ADDR="$MASTER_ADDR

cd /p/scratch/ccstdl/gordon2/open_clip
export PYTHONPATH="$PYTHONPATH:$PWD/src"
srun python -u src/training/main.py \
    --save-frequency 1 \
    --zeroshot-frequency 1 \
    --report-to tensorboard \
    --train-data="/p/scratch/ccstdl/gordon2/CC3M/train/{00000..03318}.tar" \
    --warmup 10000 \
    --batch-size=8 \
    --lr=5e-4 \
    --wd=0.1 \
    --epochs=1 \
    --workers=23 \
    --model ViT-B/32 \
    --name "32nodes" \
    --dist-url="env://"

What other things should I be doing in my script (slurm or python) to ensure that these ranks are properly connecting?
Happy to provide more details if needed :)!