Solved by Yanli_Zhao in post #3 
              
              
                @richguybobby NCCL supports point to point send/recv since 2.7.3 version, we added send and recv API on top of it in PT 1.8.0.  The NCCL operations should be async, to synchronize the stream, do you want to try to call cudaDeviceSynchronize() to synchronize all streams with the host device, or call …