Would it be possible to come up with a minimal script that reproduces the issue that we can run to reproduce it on our end? Any logs you may also have would be helpful.
Also - which distributed comm. backend are you using (Gloo, NCCL, MPI)? If all ranks indeed call into the second barrier but none make it out, I’m guessing there is likely a hang or some other form of desynchronization going on.