Hey Can,

pytorch version 1.8.1-cu102
the instance is kubeflow notebook server
container image is ubuntu:20.04
behavior is reproducible

I fixed the issue by setting master ip to localhost
export NCCL_SOCKET_IFNAME =lo
I haven’t tried a multi-node training with several GPUs per node yet.