Hi, I used similar code like this:
    for i in range( self._epoch+1, self._niter+1 ):
        if self._train_sampler is not None:
            self._train_sampler.set_epoch(i)
        self._train()           
        if self.rank==0:
            print('rank {} go to validation'.format(self.rank))
            self._validate()
        if self.distributed:
            print('rank {} go to barrier'.format(self.rank))
            dist.barrier()
            print('rank {} go out of barrier'.format(self.rank))

The output of the code is like:
rank 1 go to barrier
Training…
rank 0 go to validation
start to validate
evaluating…
rank 0 go to barrier
rank 0 go out of barrier
Then it just stopped and rank 1 won’t go out. There is no other error, just freezed. Do you have any idea what happened? When I remove the validation, the code is working. My test dataloader didn’t use the distributedSampler.