Hi @ptrblck, thanks for the suggestion.
I’ve tried to filter out the output of the NCCL info as much as possible (and removed server name / IP):
MYSERVER:84140:84140 [0] NCCL INFO Bootstrap : Using ens8f0:000.00.000.000<0>
MYSERVER:84140:84140 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
MYSERVER:84140:84140 [0] NCCL INFO NET/IB : No device found.
MYSERVER:84140:84140 [0] NCCL INFO NET/Socket : Using [0]ens8f0:000.00.000.000<0>
MYSERVER:84140:84140 [0] NCCL INFO Using network Socket
NCCL version 2.10.3+cuda11.3
MYSERVER:84140:84275 [0] NCCL INFO bootstrap.cc:107 Mem Alloc Size 28 pointer 0x7f8fa8000b20
MYSERVER:84140:84276 [0] NCCL INFO init.cc:260 Mem Alloc Size 18872 pointer 0x7f8fa0002f70
MYSERVER:84140:84276 [0] NCCL INFO misc/utils.cc:30 Mem Alloc Size 12 pointer 0x7f8fa000d390
MYSERVER:84141:84141 [1] NCCL INFO Bootstrap : Using ens8f0:000.00.000.000<0>
MYSERVER:84141:84141 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation

MYSERVER:84141:84277 [1] NCCL INFO === System : maxWidth 24.0 totalWidth 24.0 ===
MYSERVER:84141:84277 [1] NCCL INFO CPU/1 (1/2/-1)
MYSERVER:84141:84277 [1] NCCL INFO + PCI[24.0] - GPU/A1000 (0)
MYSERVER:84141:84277 [1] NCCL INFO + PCI[24.0] - GPU/C1000 (1)
MYSERVER:84141:84277 [1] NCCL INFO + PCI[3.0] - NIC/C2000
MYSERVER:84141:84277 [1] NCCL INFO ==========================================
MYSERVER:84141:84277 [1] NCCL INFO GPU/A1000 :GPU/A1000 (0/5000.000000/LOC) GPU/C1000 (2/24.000000/PHB) CPU/1 (1/24.000000/PHB)
MYSERVER:84141:84277 [1] NCCL INFO GPU/C1000 :GPU/A1000 (2/24.000000/PHB) GPU/C1000 (0/5000.000000/LOC) CPU/1 (1/24.000000/PHB)
MYSERVER:84141:84277 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 2, speed 12.000000/12.000000, type PHB/PIX, sameChannels 1
MYSERVER:84141:84277 [1] NCCL INFO  0 : GPU/0 GPU/1
MYSERVER:84141:84277 [1] NCCL INFO  1 : GPU/0 GPU/1
MYSERVER:84141:84277 [1] NCCL INFO Pattern 1, crossNic 0, nChannels 2, speed 22.000000/22.000000, type PHB/PIX, sameChannels 0
MYSERVER:84141:84277 [1] NCCL INFO  0 : GPU/0 GPU/1
MYSERVER:84141:84277 [1] NCCL INFO  1 : GPU/1 GPU/0
MYSERVER:84141:84277 [1] NCCL INFO Pattern 3, crossNic 0, nChannels 2, speed 22.000000/22.000000, type PHB/PIX, sameChannels 0
MYSERVER:84141:84277 [1] NCCL INFO  0 : GPU/0 GPU/1
MYSERVER:84141:84277 [1] NCCL INFO  1 : GPU/1 GPU/0

The rest seems to be info about memory allocation, something with “trees”, “graphs” and “rings”.
There is no other mention about NCCL failing. The last line printed is:
MYSERVER:84140:84140 [0] NCCL INFO group.cc:306 Mem Alloc Size 8 pointer 0x6814150
and the code hangs after that.
Sorry for the hassle!