Solved by wayi in post #2 
              
              
                My guess is that on Gloo backend, all the data from different GPUs need to somehow first implicitly gather the data to GPU 0 and then transfer it to CPU. 
Please not that as also summarized in Distributed communication package - torch.distributed — PyTorch 1.8.0 documentation 
This is the rule of th…