Did you install NCCL yourself or the one bundled within PyTorch?
Not sure if I interpreted the logs correctly, but looks like machine 1 is trying to use TCP while machine 2 is trying to use IB? What if you set NCCL_IB_DISABLE on both machine? https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-ib-disable 54