So did you sovle this problem?
I met the same problem while validating the trainning model during the epoch interval. It seems that dist.barrier() didn’t work.
Just as @mrshenli said:

rank 0 finishes barrier successfully but rank 1 doesn’t

Rank 0 finished validation and crossed the barrier while Rank 1 didn’t. And everything works fine after removing the validation code.
BTW, the validation was only on rank 0.