You don’t need to share the private code, a min repro would help.

But, if i remove the loss.backward() and the optimizer stuff, it works

In this case I assume you are using DistributedDataParallel (DDP)? DDP calls allreduce internally. If the first process is also in the same process group but is not running backward on DDP model, other processes would hang on backward, because they need 1st process to join the allreduce.
One requirement of collective communications is that all members need to call the same collective API in the same order.
If you need some side channel to do the barrier, you can create a new process group using the new_group 7 API, and then call barrier on that.