Solved by ptrblck in post #13 
              
              
                Thanks for the information. This points towards some communication issues between the GPUs. 
Could you run the PyTorch code using NCCL_P2P_DISABLE=1 to use shared memory instead of p2p access?