Thanks @mrshenli.
As you mentioned that pytorch has NCCL precompiled and both nodes use the same version of NCCL.
Does that mean NCCL version is not the problem?
Did you notice this “misc/ibvwrap.cc:252 NCCL WARN Call to ibv_reg_mr failed” in the logs.
I tried to build torch from source, I hit another roadblock there as well.
“Performing Test SUPPORT_GLIBCXX_USE_C99 - Failed”
thanks.