You made me open the black box  
I verified its an inside MPI implementation detail. I found it in their code.
for example openmpi 8
they use their own streams.
This is critical,
because unless the streams they create can be accessed somehow  (so far I did not find a way to do it in the cuda manual. but ill look deeper),
it means that the only way to change the behavior is editing the MPI C code and compiling.
Why do normal pytorch users should care?
Because for normal cuda-aware usage this is super-duper risky, as their streams don’t wait for the our streams, meaning cuda-aware MPI is prune to failure unless we fully synchronize our streams before each MPI call.
This would result in slower (or incorrect) program.
(I personally spent tons of time debugging this…)
I’d like to know what you think, maybe we should open an issue.
By the way, I wonder why in the file you mention irecv uses MPI_ANY_SOURCE:
is that intentional?

  
      github.com
  
  
    pytorch/pytorch/blob/f314d9a0774062a20015ae522d33eadd45293328/torch/lib/c10d/ProcessGroupMPI.cpp#L856

  auto& tensor = tensors[0];
  MPI_Request request = MPI_REQUEST_NULL;

  {
    c10::DeviceGuard guard(tensor.device());
    std::unique_lock<std::mutex> globalLock(pgGlobalMutex_);
    MPI_CHECK(MPI_Irecv(
        tensor.data_ptr(),
        tensor.numel(),
        mpiDatatype.at(tensor.scalar_type()),
        MPI_ANY_SOURCE,
        tag,
        pgComm_,
        &request));
  }

  return std::make_shared<AsyncWork>(tensor, request);
}

std::shared_ptr<ProcessGroup::Work> ProcessGroupMPI::barrier(
    const BarrierOptions& opts) {