Good observation, @ptrblck! TF most likely uses a system-wide CUDA runtime - at least it did last time I checked.
USE_TF=0 should prevent TF from loading in the current incarnation of transformers, it’s very possible that this was at least a partial culprit.
The issue got fully resolved using NCCL_SOCKET_IFNAME=lo as reported here 51 based on this thread 14.