Gloo doesn’t support reduce operation. How to use tcp backend to transfer parameters through dist.reduce and dist.broadcast? For the GPU tensor.
    for param in model.parameters():
        param.data = param.data.cpu()
        dist.reduce(param.data, dst=0, op=dist.reduce_op.SUM, group=group)
        if dist.get_rank()==0:
            param.data = param.data / size
        dist.broadcast(param.data, src=0, group=group)
        param.data = param.data.cuda()

I use the code like this, but it cost a lot of cpu, which is slower than before.
Really appreciate for any help!
Any one have good idea?