That’s really cool! Thanks for your sharing!
While I am not sure how the package works and whether it can be applied to such problem:
for iteration, data0, data1 in enumerate(data_loader, start_iter):
    tensor = model(data0)
    synchronize()
    tensors = dist.all_gather(tensor)
    loss = model(data1, tensors)

So in each process different data0 will generates a tensor, and the gathered tensors will be used for further training. Since ‘all_gather’ cannot preserve the ‘grad_fn’, can you give me some advice to solve it?
Thanks a lot.