Thank you again. one final question, I write a simple example  to see how the grad_fn works:
    # in each process:
    a = torch.tensor([1.0, 3.0], requires_grad=True).cuda()
    b = a + 2 * dist.get_rank()
    # gather
    bs = [torch.empty_like(b) for i in range(dist.get_world_size())]
    bs = diffdist.functional.all_gather(bs, b)
    # loss backward
    loss = (torch.cat(bs) * torch.cat(bs)).mean()
    loss.backward()
    print(a.grad)

I think a should has its gradient?  But currently it is None. I am a little bit lost.