One of the problems is that MPI backend doesn’t really support CUDA. Apparently we’re missing some checks for this, but it should be easy to add CUDA support. Thanks for the feedback.
Can you please provide us some more details like the script you are using for training (doesn’t have to be the same exact script, just something we can use to benchmarks) and the scripts you used for benchmarking bandwidth?
One of the reasons why you can see it running slower is this piece 36. You can see that MPI doesn’t allow us to run all-reduce in-place, so we have to do this extra memcpy that will lower the bandwidth. Additionally, the memcpy is incorrect if tensor is CUDA