Solved by stas in post #7 
              
              
                Good observation, @ptrblck! TF most likely uses a system-wide CUDA runtime - at least it did last time I checked. 
USE_TF=0 should prevent TF from loading in the current incarnation of transformers, it’s very possible that this was at least a partial culprit. 
The issue got fully resolved using NCCL…