Solved by pietern in post #2 
              
              
                The question marks mean that it depends whether or not your MPI distribution is compiled with CUDA support or not. If it is, send/recv of GPU tensors works. If it doesn’t, you’ll have to copy GPU tensors to CPU before you can pass them to send/recv. (see torch.Tensor.cpu).