The all_gather call here does look okay to me. Could you share a minimal repro script that we can try on our end? Few possibilities leading to a freeze could involve:

The order of allgather calls are not the same on all ranks.
There could be some sort of cuda synchronization causing a deadlock with allgather (ex: allgather on node1 waiting for node2, but allgather on node2 is waiting for a cuda synchronize or something).