stas:

But reading his last follow up, once he matched cuda versions of pytorch and system-wide one the basic launcher now works. Which is odd that he needed to match the two, as pytorch’s distributed shouldn’t be impacted by the system-wide cuda install.


If you’ve installed the PyTorch binaries, the shipped CUDA runtime should be used by NCCL.
However, based on the output it seems that TF is again loaded (which created other issues in the past, but we never got an update from TF):
2021-02-18 19:28:13.170701: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1

Do you know, if TF is using the locally installed CUDA runtime? If so, I would assume that you need to install PyTorch with the matching CUDA runtime if TF also imports the system-wise installation.