Does handle.wait() block and synchronize all processes like torch.distributed.barrier()? I have having trouble understanding wait and barrier`. Based on the description in torch documentation, I thought wait would also synchronize? Am I misunderstanding? I need to synchronize the results in all processes before continuing. In particular, what are the differences between the 3 blocks of code below.

x = func()
handle = torch.distributed.all_reduce(x, op=torch.distributed.ReduceOp.SUM, async_op=True)
handle.wait()
x = func()
handle = torch.distributed.all_reduce(x, op=torch.distributed.ReduceOp.SUM, async_op=True)
handle.wait()
torch.distributed.barrier()
x = func()
torch.distributed.all_reduce(x, op=torch.distributed.ReduceOp.SUM, async_op=False)
torch.distributed.barrier()