For anyone who comes up against this issue - for me, of the connections reported by ifconfig on each device, only one ip matches the devices host name. Forcing that device in NCCL_SOCKET_IFNAME has fixed that, although runs are no longer consistent - with a moderate probability chance NCCL reports

RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:761, internal error, NCCL version 2.7.8
ncclInternalError: Internal check failed. This is either a bug in NCCL or due to memory corruption

That it works at all (even with a little bit of management) is an improvement, but the stability is definitely an issue. Also, on repeated tests it appears that including the change to NCCL_SOCKET_IFNAME works for multi-GPU, multi-node, fails with a moderate probability on multi-node single-GPU, and fails with high probability on single-node single-GPU.
Likely a system issue, but rather a weird one.