seliad:

By the way, I wonder why in the file you mention irecv uses MPI_ANY_SOURCE :
is that intentional?


I am not aware of the history here. @pietern and @teng-li would know more.

Why do normal pytorch users should care?
Because for normal cuda-aware usage this is super-duper risky, as their streams don’t wait for the our streams, meaning cuda-aware MPI is prune to failure unless we fully synchronize our streams before each MPI call.
This would result in slower (or incorrect) program.

I agree, full synchronization is not acceptable here. Can MPI take a CUDA stream as an argument and then work on that stream like NCCL does? If this is possible we can let ProcessGroupMPI manage the streams and use CUDA event to synchronize.